Arizona Department of Education Arizona LEARNS Achievement Profiles Technical Manual Updated January 30, 2003 Version 1.0 Published by Arizona Department of Education.  2002 Arizona Department of Education Research & Policy All rights reserved. Printed in the United States of America. No part of this document may be reproduced or transmitted in any form or by any means without the prior written approval of the Arizona Department of Education. Table of Contents LIST OF TABLES AND FIGURES........................................................................... IV ARIZONA LEARNS ACHIEVEMENT PROFILES................................................. 1 REVIEW OF THE ORIGINAL LANGUAGE OF PROPOSITION 301 (ARS § 15-241) .............. 1 Achievement Profile Components ............................................................................ 1 School Performance Requirements.......................................................................... 1 School Classifications.............................................................................................. 1 School consequences................................................................................................ 1 OVERVIEW OF THE REVISED ACHIEVEMENT PROFILES (ARS § 15-241)....................... 2 Revising the Achievement Profiles........................................................................... 2 Collaborative Effort with districts and schools ....................................................... 2 Key Components of the Revised Achievement Profiles............................................ 2 Future Improvements to the AZ LEARNS methodology .......................................... 4 STATE BOARD APPROVAL OF AZ LEARNS ACHIEVEMENT PROFILE METHODOLOGY ........................................................................................................ 5 SUBJECT/GRADE VALUE CLASSIFICATION SCALES ....................................................... 6 Compensatory Scale Model ..................................................................................... 6 Determining Scale Cut Points for School Classification......................................... 7 ACHIEVEMENT PROFILE VERIFICATION PROCESS...................................... 8 COMMUNICATION TO DISTRICTS AND SCHOOLS............................................................ 8 Communication to districts and charter schools ..................................................... 8 Achievement Profile Verification Timeline.............................................................. 9 ELEMENTARY SCHOOL MODEL (GRADES K-8)............................................. 12 HIGH SCHOOL MODEL .......................................................................................... 15 The High School Super Cohort .............................................................................. 16 Accountability for 9th Grade .................................................................................. 16 ARIZONA INSTRUMENT TO MEASURE STANDARDS ................................... 18 OVERVIEW .................................................................................................................. 18 DETERMINING CUT POINTS FOR AIMS ....................................................................... 18 Baseline Groupings: Y Axis ................................................................................... 18 Change Point Groupings: X Axis........................................................................... 20 DETERMINING CHANGE POINTS FOR A SCHOOL’S SUBJECT/GRADE COMBINATION .... 21 ADDED EVIDENCE: MEASURE OF ACADEMIC PROGRESS AND EXTENDED WRITING.............................................................................................. 23 OVERVIEW .................................................................................................................. 23 MEASURE OF ACADEMIC PROGRESS ........................................................................... 23 Achievement Profile Technical Manual Arizona Department of Education•i Longitudinal Matching........................................................................................... 24 Making One Year’s Growth ................................................................................... 25 Using MAP in the Achievement Profile Calculation ............................................. 26 EXTENDED WRITING ................................................................................................... 28 Using EW in the Achievement Profile Calculation............................................... 28 DETERMINING SUBJECT/GRADE VALUES ...................................................... 30 DISTRIBUTING SUBJECT/GRADE VALUES.................................................................... 30 7 points................................................................................................................... 30 5 Points .................................................................................................................. 30 3 Points .................................................................................................................. 30 0 Points .................................................................................................................. 31 GRADUATION AND DROPOUT RATES............................................................... 32 OVERVIEW .................................................................................................................. 32 Rounding ................................................................................................................ 32 GRADUATION RATES .................................................................................................. 33 Background ............................................................................................................ 33 Cut-points............................................................................................................... 34 Meeting the Target................................................................................................. 34 DROPOUT RATES......................................................................................................... 34 Background ............................................................................................................ 34 Cut-points............................................................................................................... 35 Meeting the Target................................................................................................. 36 POINT VALUES AND CATEGORY OUTCOMES ............................................................... 37 7 Points .................................................................................................................. 37 5 Points .................................................................................................................. 37 3 Points .................................................................................................................. 38 0 Points .................................................................................................................. 39 Alternative Calculation.......................................................................................... 39 Summary ................................................................................................................ 40 SMALL SCHOOL SCORE ADJUSTMENT............................................................ 42 CRITERIA .................................................................................................................... 42 METHOD ..................................................................................................................... 42 EXEMPTED SCHOOLS ............................................................................................ 44 OVERVIEW .................................................................................................................. 44 Extremely Small Schools........................................................................................ 44 Alternative Schools ................................................................................................ 46 Alternative School Application Process................................................................. 48 Accommodation Schools ........................................................................................ 49 New Schools ........................................................................................................... 49 K-2 Schools ............................................................................................................ 50 PUTTING IT ALL TOGETHER: HOW TO CALCULATE AN ACHIEVEMENT PROFILE...................................................................................................................... 51 CALCULATING AN ELEMENTARY SCHOOL ACHIEVEMENT PROFILE ............................ 51 Step 1: Determine Baseline Grouping ................................................................... 51 Achievement Profile Technical Manual Arizona Department of Education•ii Step 2: Determine Growth Points for AIMS .......................................................... 51 Step 3: Determine Growth Points for Added Evidence ......................................... 53 Step 4: Determine Total Growth Points................................................................. 55 Step 5: Determine the Appropriate Point Outcome Grid ...................................... 56 Step 6: Determine Total Point Outcomes .............................................................. 57 Step 7: Determine Subject/Grade Value Scale ...................................................... 58 CALCULATING A HIGH SCHOOL PROFILE .................................................................... 58 Step 1: Determine Baseline Grouping ................................................................... 58 Step 2: Determine Growth Points for AIMS .......................................................... 58 Step 3: Determine the Appropriate Point Outcome Grid ...................................... 60 Step 4: Determine the Outcome Values for Dropout and Graduation Rate .......... 61 Step 5: Determine Total Point Outcomes .............................................................. 63 LIST OF APPENDICES ............................................................................................. 64 Achievement Profile Technical Manual Arizona Department of Education•iii List of Tables and Figures FIGURE 1: LINEAR MODEL OF MAKING OYG..................................................... 12 FIGURE 2: CONCEPTUAL MODEL OF MAKING OYG ......................................... 13 TABLE 1: HIGH SCHOOL COHORT MODEL.......................................................... 16 TABLE 2: RANK ORDERING SCHOOLS AND PAIRING ENROLLMENT COUNTS........................................................................................................................ 19 TABLE 3: DETERMINING SCORES BASED ON PERCENTAGE OF STATE ENROLLMENT ............................................................................................................ 19 TABLE 4: BASELINE GROUPINGS FOR GRADE 3 MATH ................................... 20 FIGURE 3: CHANGE POINT GROUPINGS BASED ON THE STATE DISTRIBUTION OF SCHOOL CHANGE POINTS.................................................... 20 FIGURE 4: CALCULATING A 3 YEAR AVERAGE FOR %FFB AND MEET/EXC ........................................................................................................................................ 21 FIGURE 5: POSITIVE GROWTH POINTS................................................................. 22 FIGURE 6: SUMMARY OF MAP CHARACTERISTICS .......................................... 24 TABLE 5: DETERMINING ONE YEAR’S GROWTH............................................... 26 FIGURE 7: CALCULATING A 3 YEAR AVERAGE FOR MAP .............................. 26 TABLE 6: ADDED EVIDENCE GROWTH POINTS FOR MAP............................... 27 FIGURE 8: CALCULATING A 3 YEAR AVERAGE FOR EW................................. 28 TABLE 7: DISTRIBUTION OF PERCENT OF STUDENTS WITH 24 OR MORE TRAIT SCORE POINTS............................................................................................... 29 TABLE 8: ADDITIONAL GROWTH POINTS BASED ON EXTENDED WRITING ........................................................................................................................................ 29 FIGURE 9: DISTRIBUTION OF SUBJECT/GRADE VALUES ................................ 30 FIGURE 10: GRADUATION RATE CALCULATION .............................................. 33 Achievement Profile Technical Manual Arizona Department of Education•iv FIGURE 11: CALCULATING A TWO YEAR AVERAGE FOR GRADUATION RATE ............................................................................................................................. 33 FIGURE 12: CALCULATING AN ANNUAL DROPOUT RATE.............................. 35 FIGURE 13: CALCULATING A THREE YEAR AVERAGE FOR DROPOUT RATE ........................................................................................................................................ 35 TABLE 9: TARGET IMPROVEMENTS FOR DROPOUT AND GRADUATION RATES IN THE HIGH SCHOOL ACHIEVEMENT PROFILE.................................. 36 FIGURE 14: GRADUATION AND DROPOUT CATEGORY OUTCOMES FOR A POINT TOTAL OF 5..................................................................................................... 38 FIGURE 15: GRADUATION AND DROPOUT CATEGORY OUTCOMES FOR A POINT TOTAL OF 3..................................................................................................... 39 FIGURE 16: GRADUATION AND DROPOUT CATEGORY OUTCOMES FOR A POINT TOTAL OF 0..................................................................................................... 39 TABLE 10: POINT VALUE OUTCOMES FOR DROPOUT AND GRADUATION RATES........................................................................................................................... 40 TABLE 11: POINT VALUE OUTCOMES FOR DROPOUT RATE: ALTERNATIVE CALCULATION ........................................................................................................... 41 TABLE 12: GRADE SCHOOL XA ............................................................................... 45 TABLE 13: GRADE SCHOOL X................................................................................. 46 FIGURE 17: CALCULATING A 3 YEAR AVERAGE FOR %FFB AND MEET/EXC ........................................................................................................................................ 52 FIGURE 18: POSITIVE GROWTH POINTS............................................................... 52 FIGURE 19: CALCULATING A 3 YEAR AVERAGE FOR MAP ............................ 54 TABLE 14: ADDED EVIDENCE GROWTH POINTS FOR MAP............................. 54 FIGURE 20: CALCULATING A 3 YEAR AVERAGE FOR EW............................... 55 TABLE 15: ADDITIONAL GROWTH POINTS BASED ON EXTENDED WRITING ........................................................................................................................................ 55 FIGURE 21: CALCULATING A SCHOOL’S TOTAL CHANGE POINTS FOR GRADE 3 MATH .......................................................................................................... 56 TABLE 16: SUBJECT/GRADE VALUES GRID ........................................................ 56 FIGURE 22: EXAMPLE: MATH GRADE 3 BASELINE GROUPING AND GROWTH CUT POINTS .............................................................................................. 57 Achievement Profile Technical Manual Arizona Department of Education•v FIGURE 23: POINT VALUE OUTCOMES FOR A K-8 SCHOOL............................ 57 FIGURE 24: NINE SUBJECT/GRADE VALUES SCALE ......................................... 58 FIGURE 25: CALCULATING A 3 YEAR AVERAGE FOR %FFB AND MEET/EXC ........................................................................................................................................ 59 FIGURE 26: POSITIVE GROWTH POINTS............................................................... 59 TABLE 17: SUBJECT/GRADE VALUES GRID ........................................................ 60 FIGURE 27: EXAMPLE: MATH GRADE 10 BASELINE GROUPING AND GROWTH CUT POINTS .............................................................................................. 61 TABLE 18: POINT VALUE OUTCOMES FOR DROPOUT AND GRADUATION RATES........................................................................................................................... 62 TABLE 19: POINT VALUE OUTCOMES FOR DROPOUT RATE: ALTERNATIVE CALCULATION ........................................................................................................... 63 FIGURE 28: HIGH SCHOOL FOUR SUBJECT/GRADE VALUES.......................... 63 FIGURE 29: FOUR SUBJECT/GRADE VALUE SCALE .......................................... 63 Achievement Profile Technical Manual Arizona Department of Education•vi Arizona LEARNS Achievement Profiles Review of the original language of Proposition 301 (ARS § 15-241) November 2001 Arizona voters approved Proposition 301(Prop. 301), which provided both increased teacher pay as well as public accountability for school performance. A summary of the language adopted in Prop. 301, the original legislation, is provided for reference below. For a more detailed explanation of this proposition please refer to the full legislation. Arizona Instrument to Measure Standards (AIMS) scores and passage rates Achievement Profile Components Adequate yearly progress (MAP) and Dropout rate School Performance Requirements 90% of students Meet or Exceed the Standards on Arizona’s Instrument to Measure Standards OR a percentage higher than the previous year and 90% of students progress one year or more during the school year (Measure of Academic Progress) OR a percentage higher than the previous year and Dropout rate of 6% OR a percentage lower than the previous year. (A percentage is interpreted as 1.0% increase or decrease over the previous year) School Classifications Underperforming- if a school fails to annually produce acceptable progress in each area Failing- if a school remains underperforming in same achievement area for a second year School consequences An underperforming school must have its board notify each residence in attendance area, develop improvement plan for school, Achievement Profile Technical Manual Arizona Department of Education•1 and present improvement plan at special meeting at underperforming school A failing school must follow all of the above, plus ADE assigns to the school a “trouble solutions team” to work with the staff, publishes semi-annual list of all failing schools, and students may choose alternative tutoring program in academic standards. Overview of the Revised Achievement Profiles (ARS § 15-241) Revising the Achievement Profiles The Research and Policy Division (R&P) of the Arizona Department of Education (ADE) began the process of restructuring the Achievement Profile methodology after a preliminary analysis revealed a number of flaws in the existing profiles as outlined in Proposition 301. Given the nature of these flaws, it was necessary to go beyond the limitations of the adopted profiles. Collaborative Effort with districts and schools R&P consulted a diverse group of experts, ranging from measurement experts, curriculum coordinators to classroom teachers. These experts volunteered their time to undertake the difficult task of advising R&P on the complex issue of state-level school accountability. During the summer of 2002 the ADE met with district and educational representatives and formed the AZ LEARNS/NCLB Accountability Formula Working Group. This group worked to create the formula for AZ LEARNS, the baseline groupings, change points, as well as discussed the inclusion of mobile students and English Language Learners into the state’s accountability system. This group dealt with the fundamental questions regarding school accountability and sought to develop a fair and accurate system to measure student achievement and school performance. The ultimate purpose of the revised Achievement Profile was to advance student learning at the local level. This purpose guided all decisions in the development process. With that understanding, the ADE along with the AZ LEARNS/NCLB Accountability Formula Working Group developed AZ LEARNS with the explicit purpose of identifying schools using a viable classification system based on a set of academic performance indicators. Key Components of the Revised Achievement Profiles The following are key components of the revised AP that are necessary to meet the purpose and ensure the revised AP are both sound and effective. A spectrum of school “classifications” to identify diverse school outcomes. School performance is based on a continuum, and multiple school classifications are necessary to most accurately reflect that continuum. In addition, a multiple classification system is the optimal method to provide meaningful information to stakeholders. Achievement Profile Technical Manual Arizona Department of Education•2 The revised AP will include classifications to identify diverse states of school performance. Absolute and contextual achievement indicators to report school performance in the appropriate context. Contextual achievement indicators are established through an empirical process. The results of this process yield cut scores, which are grounded within the context of student achievement in our state. An empirical process is the most accurate approach, because the scores will be based solely on the results of data analysis. Our goal is to classify every school accurately. Any school-level scores that are established without the proper consideration of all necessary data are essentially arbitrary, resulting in the likely misclassification of schools. An achievement system for elementary and high schools Longitudinal data to capture student and school trends Academic achievement by all student groups Consideration of the effects of student mobility Multiple outcome indicators, such as dropout, attendance and graduation rates Multiple levels of reporting, ranging from the media level to the school/classroom level. Improved student learning is the central goal of the revised AP. To that end, the results of the revised AP must be communicated at multiple levels and with varied amounts of detail depending on the target audience. At the media level, revised AP results will be “publicly consumable” to allow for the information to be communicated efficiently and easily understood by a diverse audience. As the target audience approaches the classroom level, the level of detail will increase to provide information for programmatic and instructional purposes. R&P will ensure that each level builds on the previous level and provides consistent information. Development of one comprehensive system to supersede other fragmented accountability measures in state statute and to fulfill federal requirements A parallel accountability system for unique schools (i.e., accommodation, extremely small). In order to avoid the pitfalls of a “one size fits all” approach, the revised AP include a parallel accountability system for schools with unique characteristics. The list of unique schools includes accommodation and extremely small schools. The unique characteristics of this subset of schools preclude them from fitting into a general accountability system with fair treatment. Many states provide a parallel accountability Achievement Profile Technical Manual Arizona Department of Education•3 system for these schools with the goal of improving the accountability system for all schools. Future Improvements to the AZ LEARNS methodology It should be noted that the current AZ LEARNS model was developed as a transition plan to comply with Arizona state statute. The AZLEARNS/NCLB Accountability Formula Working Group intended to address specific aspects of State legislation, rather than to fully integrate the AZ LEARNS model with the No Child Left Behind Act of 2001 (NCLB). The Arizona Department of Education (ADE) understands that the AZ LEARNS model must comply with the requirements of the NCLB legislation. The model, therefore, can be modified or adapted to accommodate future improvements or deal with currently unresolved issues. Some of these unresolved issues, which will need to be addressed in the future, include, but are not limited to, disaggregating student level data by subgroup populations; the inclusion of English language learners (ELL) and special education students; and developing NCLB annual measurable objectives. These issues will not be addressed in this technical manual; however, it is important to note that they have not been overlooked and will be resolved in the future. Achievement Profile Technical Manual Arizona Department of Education•4 State Board Approval of AZ LEARNS Achievement Profile Methodology On September 23, 2002 the Arizona State Board of Education met to review, comment and approve the AZ LEARNS methodology, or formula, and the subsequent components of the Elementary and High school accountability models. The ADE provided the State Board an Information packet which outlined the decisions regarding the formula which needed to be made, the attachments which were required to facilitate these decisions, a list of Board motions for AZ LEARNS as well as the attachments which were required to pass these motions. This documentation can be found online via the AZ LEARNS web site, http://www.ade.az.gov/azlearns/ or by referring to the appendix. The Board reviewed and approved the following motions, which are germane to the Achievement Profile methodology/formula; these can be found in the appendices at the back of this document, they are numbered in accordance to the State Board action item numbers assigned on September 23, 2002: AZL1: Adoption of Baseline Group Separation Points, see Appendix 1. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A1-BaselineGroupings2000.pdf AZL2: Adoption of the Added Evidence Growth Points for MAP, see Appendix 2. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A2-AddedEvidenceMAP.pdf AZL3: Adoption of the Added Evidence Growth Points for Extended Writing, see Appendix 3. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A3-AddedEvidenceWriting.pdf Achievement Profile Technical Manual Arizona Department of Education•5 AZL4: Adoption of the Growth Cut Points for each subject/grade combination, see Appendix 4. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A4-GPCutPoints.pdf AZL5: Adoption of the Graduation and Dropout Rate targets, see Appendix 5. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A5-GradDOTargetsValues.pdf AZL6: Adoption of the mechanics of the subject/grade value table, see Appendix 6. AZL7: Adoption of the definition, criteria and timeline for the alternative school Achievement Profile, see Appendix 7. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A7-AltvSchDefinition.pdf AZL8: Adoption of definition, criteria and timeline for the small school Achievement Profile see Appendix 8. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A8-APSmSchAltvSch.pdf AZL13: Adoption of the subject/grade value classification scalesall grades, see Appendix 9. To view the online version, click on the following link: http://www.ade.az.gov/azlearns/A9-CompensatoryModels.pdf AZL14: State Board information packet. For an overview, see Appendix 10 of this document, for the complete version; please visit the AZ LEARNS web page at http://www.ade.az.gov/azlearns/Board9-23submitted.pdf Subject/Grade Value Classification Scales Compensatory Scale Model The ADE recommended to the State Board that they adopt school classification scales that are consistent with a compensatory model. In a compensatory approach, higher subject/grade values can have enough influence to improve the school classification despite the presence of low subject/grade values. The school classification scales were developed based on the State Board’s decision in two pivotal cases. The first pivotal case involves breaking ties in cases where there are an equal number of adjacent subject/values. The second pivotal case involves the extreme scenario where a few number of the highest subject/grade value (7) can compensate for a larger number of the lowest subject/grade values (0). The Board decided to break ties using the compensatory method while still setting minimum expectations for anomalous extreme high values, which should not offset consistently low subject/grade values. Achievement Profile Technical Manual Arizona Department of Education•6 The State Board approved eight (8) Compensatory Subject/Grade Value scales. These subject/grade value scales are necessary to capture the wide range of grade configurations allowed in the state. For example, a school with grades K-8 would be evaluated based on a nine (9) subject/grade scale because there are 3 grades assessed (3, 5, and 8) each using 3 subject areas (mathematics, reading and writing). Determining Scale Cut Points for School Classification These scales were first determined using the range of scores for each subject/grade combination, with possible values being 0, 3, 5 and 7. Therefore, the range for the 9 subject/grade value scale is 063. Zero represents the lowest obtainable value and 63 represents the highest obtainable value in this scale. Each of the individual scales evaluates a school based on this methodology. The ADE determined the cut-points for each of the scales at the maintaining and improving demarcations using the following methodology: 1. Each possible combination, or permutation, of the values in each of the scales was developed based on the range of scores possible in each scale. This resulted in a range of 10 permutations in the 2 Subject/Grade Value scale to 220 permutations in the 9 Subject/Grade Value scale. 2. In all subject/grade scales, the score consistent with Maintaining, all subject/grade values of 3 on the scale, must fall within the range of Maintaining [adequate performance]. 3. In even number subject/grade values (2, 4, 6, and 8) in the event of a tie with adjacent values two 3 values and two 5 values, the cut-point is set so that the benefit of the doubt is given to the school and they fall into the higher classification. 4. In odd number subject/grade values (3, 5, 7, and 9) the weight of 2/3 of an individual value can offset lower values. 5. In 2% of the subject grade values, or 15 instances, the scales developed did not accurately reflect school performance. In these cases, the State Board adopted exceptions to the scales. Primarily, the exceptions set minimum values, which are used to designate a school maintaining. In the scales where the majority of the values are zero (0), but a combination of remaining scores places a school in the Maintaining range the State Board determined that the appropriate classification should be Underperforming. This affected the 4, 5, 6, 7, 8 and 9 subject/grade value scales. These exceptions to the scales can be found in the Attachment 9. Achievement Profile Technical Manual Arizona Department of Education•7 Achievement Profile Verification Process Communication to Districts and Schools Due to the nature of the Achievement Profiles as outlined in ARS § 15-241, AZ LEARNS, it was imperative that the ADE make attempts to communicate to schools their AP status prior to the public release to verify its accuracy. An integral component in this process involved the verification of schools’ data. This verification process became the basis to substantiate the final AP for each school. The following will provide the timeline, which was used by the ADE to facilitate the verification process. Due to the number and length of the memos, which were used to notify and update schools regarding the AP process these memos will not be attached in this document, however, they are available upon request. It is important to note that in addition to direct communication to districts and charter holders, the ADE developed a web page for districts and schools to obtain updated information regarding AZ LEARNS verification and public release data as well as other related information. This is a site that can be accessed publicly and will continue to be used to update schools and districts about future notifications regarding AZ LEARNS. The site can be accessed at www.ade.az.gov/azlearns/. Communication to districts and charter schools During the course of the summer of 2002 the ADE began communication with districts Superintendents and charter school holders regarding the AP. During this time an Achievement Profile email account was created, with an AZ LEARNS contact for each district and charter holder. This person, once identified received all subsequent information. The Achievement Profile email account continues to be maintained, for districts and charter holders to communicate their questions to ADE staff. In addition, memos were distributed through state agencies and private educational organizations. This list is not exhaustive; rather it represents a sample of the organizations that assisted with the dissemination of information about the Achievement Profiles: County Superintendents’ offices ADE Academic Standards and Accountability’s internal system to communicate to districts and charter schools (list serve of testing coordinators) Achievement Profile Technical Manual Arizona Department of Education•8 Charter School Board/State Board for Charters Arizona School Administrators Arizona School Board Association Greater Phoenix Education Management Council Southern Arizona Curriculum Council National Education Association/Arizona Education Association Arizona Association for School Business Officials These key organizations were asked to distribute memos pertaining to AZ LEARNS Verification process and updates to their members in order to increase the likelihood that all school Superintendents and Charter Holders would receive the necessary information. Achievement Profile Verification Timeline In order to document the verification process the date, title, recipients and a brief summary of each memo is provided below. Again, complete copies of each memo are available if requested. September 23, 2001- Achievement Profile Verification, this memo was sent to all District Superintendents and Charter School Administrators. In order to accurately report for all schools, local administrators must verify the following information: all schools in the district or under the charter that are expected to receive an Achievement Profile on October 15, 2002, the grades offered at each school for the 1999-2000, 2000-2001 and 2001-2002 academic years. October 2, 2002 is the deadline to complete this process. If a district or charter holder fails to complete the verification process the ADE will assume that the schools on file and their subsequent data are complete and correct, therefore the ADE will produce and Achievement Profile for these schools. October 1, 2002- Missing Achievement Profile Verification, this memo was sent to all District Superintendents and Charter School Administrators who had yet to complete the verification process. As of September 30, 2002 our records show that your district or charter holder has not verified Achievement Profile information. This information must be verified by Friday, October 4, 2002. Achievement Profile verification is available to administrators through the common logon. October 9, 2002- Preliminary Achievement Profile Verification, this memo was sent to all District Superintendents and Charter School Administrators. On Thursday, October 10, 2002 the Achievement Profile Verification Application will reopen to allow you or your school designee to view your schools’ preliminary Achievement Profiles. Achievement Profile Technical Manual Arizona Department of Education•9 Where applicable, schools will be allowed one final opportunity to account for any missing data via the online application. In order to process this information the application must shut down at midnight October 13th, you will not be able to view the profile information on October 14th. The public release of the Achievement Profiles will take place on Tuesday, October 15th. October 15, 2002- Achievement Profile Public Release, Achievement Profiles were released via the AZ LEARNS web site for each school in the state, with the following exceptions: New Schools Alternative School Applicants Extremely Small Schools Pending Schools and K-2 Schools October 18, 2002- Preliminary Achievement Profiles—Schools that did not receive a Profile on October 15, 2002, this memo was sent to District Superintendents and Charter School Administrators. There are a number of schools that did not receive an Achievement Profile on October 15, 2002 because they fell into one or more of the following categories: Alternative School Applicants Extremely Small Schools and Pending Schools These schools will receive an Achievement Profile on November 1, 2002. Beginning Tuesday, October 29, 2002 the Achievement Profile application will reopen to allow districts and charter holders to view Preliminary Achievement Profiles prior to their public release on November 1, 2002. One final opportunity to verify data is available between Tuesday, October 29th and Thursday, October 31st. November 1, 2002- Public Release of all remaining Achievement Profiles were released via the AZ LEARNS web site for each school in the state, with the following exceptions, which will not receive an Achievement Profile for 2002-2003: New Schools K-2 Schools and Alternative Schools November 1, 2002- Achievement Profile School-Level Reports Available, this memo was sent to District Superintendents and Charter School Administrators. Achievement Profile Technical Manual Arizona Department of Education•10 Starting at 12 noon on Friday, November 1, 2002, detailed schoollevel reports will be available to district and charter school holder administrators through the common logon Achievement Profile Reports web site (formerly know as the Achievement Profile Verification web site). School summary reports, subject/grade combination reports as well as formula calculations (text files) and codebooks will be provided in order to provide detailed information regarding the performance of your schools. Achievement Profile Technical Manual Arizona Department of Education•11 Elementary School Model (Grades K-8) Under the revised AP, elementary schools will be evaluated based on a combination of two primary indicators: AIMS and MAP. AIMS provides a benchmark to the Arizona Academic Standards, and MAP provides a longitudinal view of individual student progress across all grade levels. Combined, these two indicators ensure that all grade levels in the elementary school share in the responsibility of moving all students forward. The AIMS component is an absolute standard that provides a snapshot of 3rd, 5th and 8th grade performance across multiple years and focuses on reducing achievement gaps between groups of students. MAP is a contextual standard that takes into consideration where individual students begin (on an academic level) and ensures all students, regardless of achievement level, are making One Year’s Growth (OYG) each academic year. Figure 1: Linear Model of Making OYG AIMS AIMS 3-year 3-year 3-year Average Average Average Grade 1 2 3 4 5 AIMS 6 7 8 MAP (SAT9) The AIMS Indicator is a “camera” pointed at 3rd, 5th and 8th grade. The classification of schools will be based on the advancement of all students to a higher performance level. School level results will be reported to the public by the percentage of students in each performance level. The purpose is to illustrate movement between performance levels. AIMS is an absolute measure of school progress. Each year a different group of students will have to demonstrate improvement over last year’s students, regardless of where they start, and all schools will be compared to each other, regardless of school Achievement Profile Technical Manual Arizona Department of Education•12 characteristics. Since all schools will be expected to reach this absolute standard, regardless of where they started, some schools will have a significantly more difficult job. Absolute improvement on AIMS will be the most challenging task for schools. Ultimately, this indicator will determine if a school is increasing the achievement levels of all students over time by measuring progress in grades 3, 5, and 8 regardless of differences between student groups. Due to the difficulty of this task, satisfactory school progress is an incremental process. Therefore, the measurement instrument must be sensitive enough to capture real school progress. Student groups vary from year to year and their performance will fluctuate from one year to the next, regardless of schooling effects. This phenomenon is known as a cohort effect. Both AIMS and MAP results will be based on a 3-year rolling average to capture trends, rather than aberrations, and to account for differences in student cohorts. A 3-year average mitigates these cohort effects and increases the reliability of the results. Furthermore, multiple years allows for the identification of real trends in school performance. A downward bump may be attributed to cohort effects, but a downward trend (continuous bumps) is an indication of lagging performance. The MAP Indicator will ensure that students in all achievement groups are making One Years Growth (OYG) based on the results of the Stanford Achievement Test, Ninth Edition (SAT9). MAP also ensures that all grades are represented in the revised AP. By requiring that students in all achievement levels make OYG, the elementary model ensures that all students are progressing academically. Another term for student’s position relative to all students in the norming sample at their particular grade is stanine. Schools will be evaluated based on the percentage of students at each stanine that achieve OYG. Figure 2: Conceptual Model of Making OYG A Stanines 1 2 B 3 4 C 5 6 7 8 9 (SAT 9) Below Average Average Above Average Though remaining at the same stanine is an accurate measure of OYG, this standard will not be sufficient for some schools to demonstrate adequate progress on AIMS. Schools with large numbers of students in the lower stanines will be required to Achievement Profile Technical Manual Arizona Department of Education•13 advance students into higher stanines in order to meet the absolute standard at the AIMS grade levels. The elementary model is designed to allow MAP to complement AIMS, but the model maintains primary emphasis on achievement of the academic standards via AIMS. Achievement Profile Technical Manual Arizona Department of Education•14 High School Model Like the elementary model, the high school model incorporates multiple measures to capture the breadth of school performance. The three measures in the high school model are the following: AIMS test score results Graduation rate -andDropout rate. Just as the elementary model, three years of AIMS results will be included into a rolling average for reading and writing results and two years of AIMS results will be included for mathematics. Due to changes in state testing the 1999-2000 mathematics results are not comparable to the 2000-01 and 2001-02 administrations of the high school AIMS mathematics assessment. Therefore a two year average will be computed for AIMS mathematics. Currently, the methodology for high schools uses all grades tested (10, 11 and 12) in each of the years provided. The current high school calculations were used as a transition to the intended high school methodology, which utilizes a cohort model to calculate a high schools’ Achievement Profile. This methodology at the high school level utilizes AIMS results, which are examined for each graduating cohort to determine how many students passed the AIMS test. A statewide, “super-cohort” of students will be established based on the students who enter 9th Grade. School performance will be based on the percentage of students in that cohort who pass AIMS by the end of 12th grade1. The following table illustrates the components and timing of the high school cohort model: 1 The Graduating Cohort Class of 2002 will be the first complete testing cohort. Therefore, an interim report will be provided based on the data available for grades 10, 11 and 12. Achievement Profile Technical Manual Arizona Department of Education•15 Table 1: High School Cohort Model Student Cohorts SY 2000 10th SY 2001 10th SY 2002 10th SY 2003 10th SY 2004 10th (Class of 2002) (Class of 2003) (Class of 2004) (Class of 2005) (Class of 2006) th 11 12th AIMS % Pass AIMS Graduation Rate % Graduates Annual Dropout Rate The High School Super Cohort All grades th 11 12th th 11 12th 11th 12th % Pass % Pass % Pass AIMS AIMS AIMS % % Graduates % Graduates Graduates All grades All grades All grades In future calculations of the Achievement Profiles, Arizona students will be treated as a Super Cohort across all schools. As members of this Super Cohort transfer from school to school, each school is responsible for the achievement of all enrolled students. To this end, Graduation Rates, Dropout Rates and AIMS scores will hold high schools accountable for all incoming transfer students, regardless of when they enter. The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. By examining a cohort of students who began high school at the same time, the graduation rate assesses how many students actually complete high school. The Dropout Rate is an annual measure of how many students drop out of a school. It is expressed as the proportion of students who had the opportunity to drop out, and did, during a 12-month reporting period. The dropout rate provides an annual snapshot and detects more immediate changes in school attendance than the graduation rate. The Graduation and Dropout Rates are important compliments to the AIMS results in the revised Achievement Profiles. Graduation Rates indicate the success of students in meeting course requirements and achieving passing grades in subject areas not covered by the AIMS test. Dropout rates are included in the high school model as a measure of student persistence and to ensure that AIMS results reflect the largest percentage of the student population possible. Accountability for 9th Grade Other measures used in the high school model, as well as measures used in the elementary model such as MAP and AIMS, are not available for 9th Grade. Schools were asked to complete a 9th grade indicator portion on their School Report Card, which was used for informational purposes only. Information provided by schools was Achievement Profile Technical Manual Arizona Department of Education•16 reported out to parents via the School Report Card, but this information was not used in the calculation of a high school’s Achievement Profile. Achievement Profile Technical Manual Arizona Department of Education•17 Arizona Instrument to Measure Standards Overview The Arizona Instrument to Measure Standards (AIMS) is a criterion referenced test used by the state of Arizona to measure student performance in the areas of math, reading and writing. In each subject area, students are grouped into performance categories based on how they performed relative to the state standard: Falls Far Below the Standard (FFB) Approaches the Standard (APP) Meets the Standard (Meet) Exceeds the Standard (EXC) The AIMS portion of the Achievement Profile carries an expectation that students will meet the state standards. For this reason, a year-to-year analysis of the percentage of students that fall far below the standard and meet or exceed the standard is used for determining a school classification. Due to variation in student performance from grade level to grade level, the expectations for improvement, cut-points, for these categories will change with each grade and subject combination. It is important to note, that only students with valid AIMS scores will be included in the analysis for the Achievement Profiles; students who have not been instructed for three (3) academic years in English, as well as students testing under non-standardized conditions are excluded from schools’ data. Determining Cut Points for AIMS Baseline Groupings: Y Axis Arizona follows the federal No Child Left Behind (NCLB) legislation in the determination of baseline groupings for each grade level. Six grouping levels are computed from a percentage of students who meet or exceed the standard in the state: the cutoffs are based on 20%, 40%, 60% and 80% of student enrollment in the state. Achievement Profile Technical Manual Arizona Department of Education•18 In order to compute these groupings, all schools in Arizona were ranked in descending order according to the percentage of students in each grade and subject combination that met or exceeded the standard on AIMS. Then, enrollment counts were paired with each school (see example in Table 2). It should be noted that the sixth baseline group is distinguished by state statue as outlined in ARS§15-241, the excelling definition, which states that 90% of students must Meet/EXC the standard in order to be excelling. Table 2: Rank Ordering Schools and Pairing Enrollment Counts Grade 3 Subject Math School School 1 School 2 School 3 School 4 School 5 School 6 School 7 School 8 Percent Meet or EXC 100 75 70 61 55 48 45 32 Enrollment 20 20 20 20 25 25 30 40 The five remaining groupings were set at 20%, 40%, 60% and 80% of students enrolled in the state. Subsequent percentages of students meeting the standard were used to set the baseline groupings for each grade and subject combination. Table 3: Determining Scores Based on Percentage of State Enrollment Grade 3 Subject Math School School 1 School 2 School 3 School 4 School 5 School 6 School 7 School 8 Percent Meet % of Total State or EXC Enrollment Enrollment 100 20 100.0% 75 20 85.0% 70 20 75.0% 61 20 65.0% 55 25 55.0% 48 20 42.5% 45 25 32.5% 32 40 20.0% For example, looking at Table 4, the 6 baseline groupings, according to the percent of state enrollment, would be the following: Achievement Profile Technical Manual Arizona Department of Education•19 Table 4: Baseline Groupings for Grade 3 Math Change Point Groupings: X Axis Grade 3 Subject Math Grouping 6 3 Year Average Meet or EXC 90-100 Cut Point ARS § 15-241 Grade 3 Subject Math Grouping 5 Percent Meet or EXC 75-100 Cut Point 75 4 61-74 61 3 2 48-60 32-47 48 32 1 0-31 Start Point The second element of the Achievement Profile requires the ADE to calculate growth points and set criteria in order to qualify student achievement gains. The growth points represent the percent change over time of students that FFB and Meet/EXC the standard, relative to the baseline. The percent change is computed by taking the difference between the three year average percent of students in each category and the baseline percentage. For example, for grade 3 math, the change points for the state are calculated by taking the difference between the three year average percent of students in each category and subtracting the baseline percentages. As in the calculation of a school’s grade/subject change points, the two differences are added together to obtain a total change point value for each subject and grade combination. Change point groupings are based on the statewide distribution of change points. Six change point groupings are computed using the state mean and the standard deviations of the distribution (see Figure 3). The 5 cut-point values associated with the change point groups are calculated separately for each subject and grade combination. Figure 3: Change Point Groupings Based on the State Distribution of School Change Points <-1.0 SD Change Point Group: 1 Achievement Profile Technical Manual State Mean -0.5 SD -1.0 SD 2 3 +0.5 SD 4 > 1.0 SD +1.0 SD 5 6 Arizona Department of Education•20 For example, the grade 3 math change point groupings and associated cut-point values were determined to be the following: <-1.0 SD Change Point Group: 1 -6.72 0.79 8.30 15.81 -1.0 SD -0.5 SD State Mean +0.5 SD 2 3 4 23.32 +1.0 SD > 1.0 SD 5 6 Determining Change Points for a School’s Subject/Grade Combination Determining a school’s change points for each subject/grade combination on the AIMS portion of a profile is based on student movement out of the FFB category and student movement into the Meet/EXC category. The value of the change points is determined by the difference between the three year average and the baseline percentages. A three year average is calculated by adding the total number of students in each category over three years and dividing by the three year total number of students tested for each subject/grade combination (see Figure 4). Figure 4: Calculating a 3 Year Average for %FFB and Meet/Exc = 3 Year Ave Total Number of Students that FFB or Meet/EXC over 3 years X 100 Total Number of Students Tested Over 3 Years High school AIMS math test does not use the academic year 2000 for the baseline year. The baseline for this subject/grade combination is 2001. The ADE considers a school to have made positive change if the three year average percentage of students that fall into the Meet/EXC category is higher than the baseline percentage or if the three year average FFB is lower than the baseline. The ADE considers a school to have made negative change if the three year Achievement Profile Technical Manual Arizona Department of Education•21 average percentage of students in the Meet/EXC category is lower than the baseline or if the FFB category is higher than the baseline percentage. The ADE expects schools to increase the percentage of students that meet the standards over time, and not increase the percentage that falls far below the standard (see Figure 5). Figure 5: Positive Growth Points FFB APP MEET/EXC For example, assume a school has the following data and enrollment is constant: Grade 3 Subject Math Academic Year 2000 2001 2002 % FFB 25.56 37.80 28.57 % Meet/Exc 34.44 32.93 35.71 3 Year Averages 30.70 34.21 The three year average for the percentage of students in the FFB category is higher than the baseline percentage, which is a negative change. Likewise, the three year average for students that met the standard is slightly lower than the baseline percentage (34.21 compared to a baseline of 34.44). This is also a negative change; a school should increase the percentage of students meeting the state standard over time. The change points associated with grade 3 math for this school will be determined according to the following: FFB Meet/EXC 3 Year Average 30.7 34.21 Baseline 25.56 34.44 Difference -5.14 -0.23 Since these changes both represent a negative change, in terms of the academic growth in this example, the values will be negative, such that the total change points in grade 3 math for this school will be -5.37. Achievement Profile Technical Manual Arizona Department of Education•22 Added Evidence: Measure of Academic Progress and Extended Writing Overview The Added Evidence portion of an elementary profile relies on the Arizona Measure of Academic Progress (MAP), which uses longitudinal Stanford 9 test scores to provide a measure of student academic growth, and Extended Writing trait scores from the AIMS test. These two items work together in order to provide additional information about the magnitude of a school’s performance; they are used as “Added Evidence” in order to supplement an elementary school’s performance level. In a way, MAP and Extended Writing scores are used as “bonus points” and can only be used to attain a higher classification; a lack of these points cannot affect the profile outcome in a negative fashion. It is important to note, that only students with valid Stanford 9 scores will be included in the analysis for the Achievement Profiles; students who have not been instructed for three (3) academic years in English, as well as students testing under non-standardized conditions are excluded from schools data. Measure of Academic Progress Unlike traditional measures of achievement, such as percentile ranks that mark achievement at one point in time, MAP measures growth over time. A measure of the progress made over a school year is obtained through linking individual student test scores from one year to the next. This progress is attributed to the school the student attended, if a student has remained in the same school for two academic years. Stanford 9 stanine scores (MAP) are only used for math and reading Added Evidence points. Added points for writing are taken from the AIMS trait scores. For more information on writing points, see Extended Writing, page 28. Achievement Profile Technical Manual Arizona Department of Education•23 Traditional indices of achievement, such as a comparison of schools’ percentile ranks calculated at a point in time, are highly correlated to student demographic variables. As a result, the same schools consistently score at the top and bottom of the percentile rank listings. With MAP as a measure of school effectiveness, schools traditionally seen as low performing, by way of a percentile rank, may show remarkable gain with the students they have had an opportunity to teach. Figure 6: Summary of MAP Characteristics MAP Key Features: Utilizes Stanford 9 stanine scores Captures individual student growth over time Accounts for mobility Includes only those students a school has had an opportunity to teach Provides meaningful information to teachers Focuses on all students Aligns to the Arizona Academic Standards in elementary reading and math Longitudinal Matching In order to determine the number of students that made One Year’s Growth (OYG), individual student records were matched from test year to test year. The process of matching Stanford 9 student records between test years involve the use of four pieces of student level data. Perfect student matches were made if all of the following were true: The student attended the same school during two years of consecutive testing The student had the same first and last name The student’s gender was the same for both years And the birth date of the student was identical for both years. However, due to inconsistencies in some of the student information (e.g. if a student misspelled his or her name, or if a mistake was made on any other piece of necessary information) matching student records that were not picked up in a perfect match involved a multi-stage process. Achievement Profile Technical Manual Arizona Department of Education•24 For example, if a student spelled his name “Frank Adams” in 2001 and misspelled it as “Grank Adams” in 2002, Frank’s records would not be matched in the initial pass. If all other pertinent pieces of information were identical, the second phase of the matching process would in all likelihood link his records. In addition to letter changes or letter transposition, the matching routine could also handle transposed birth dates. For example, if Frank Adams had also indicated that he was born 09/10/90 in 2001 and 90/10/09 in 2002 his record would likely be linked. Typically, close to 90% of student test scores can be matched between any two test years. The multistage approach to matching has limitations, however. If the student’s first or last name varied by more than two letters from one year to the next, the records cannot be matched. Also, students that do not provide their birth dates or indicate their gender in one of the years cannot be matched. In rare instances, false matches can occur between two different students in different test years. This is most likely to occur when students are related, attending the same schools in successive years and have similar first names, identical last names and are a year apart in age. Given the high proportion of test scores that are appropriately matched, the above stated limitations are of minor concern. Making One Year’s Growth MAP is used to measure individual student growth. Student Stanford 9 test scores are linked from one year to the next and growth on the test is calculated. One Year’s Growth (OYG) is broadly defined as attaining the same level of absolute achievement from year to year, while learning more difficult material. OYG is determined by examining a student’s stanine score: if one attains the same stanine score or a higher stanine score relative to the previous year, that student has made one year’s growth. There are two exceptions to this approach (see Table 5): Students who begin in stanine 9 and move to stanine 8 will make OYG Those that stay in stanine 1 from one year to the next will not make OYG For example, a student who begins at the 5th stanine (50th percentile) as a 3rd grader and maintains a 5th stanine score as a 4th grader has achieved OYG. This is the minimum growth that is expected for any student who remains at a school for an academic year. When all students achieve OYG, schools ensure that no students are falling behind from one school year to the next. Achievement Profile Technical Manual Arizona Department of Education•25 Table 5: Determining One Year’s Growth Stanine 2001 5 6 9 5 1 Stanine 2002 5 7 8 4 1 Result OYG OYG OYG Did not Accomplish OYG Did not Accomplish OYG The Stanford 9 test is distributed to elementary grades two through nine and measures student performance in the areas of math, reading and language arts. However, due to the lack of comparability of the language arts portion of the SAT-9 to state standards, only the reading and math portions are used in order to calculate MAP. Additionally, unlike the AIMS growth points, Added Evidence points are determined by the performance of the entire school. Using MAP in the Achievement Profile Calculation Math and reading Added Evidence scores are taken from a three year average of the percentage of students in each school that accomplish OYG from one year to the next. A three year average is calculated by taking the combined total number of students that make OYG and dividing by the combined total number of students tested in math and reading (see Figure 7). Figure 7: Calculating a 3 Year Average for MAP 3 Year Average Total number of students who made OYG over 3 years = Total number of students who were tested over 3 years X 100 If the three year average percent of students making OYG is greater than 60% for a subject/grade combination, growth points are attributed to each grade and subject area. The growth points are determined on the relative weight that would be given to Added Evidence in the model. To remain consistent with No Child Left Behind expectations, which require accountability to be measured using local criterion, referenced tests (AIMS) the ADE determined that the main emphasis in the model needed to reflect AIMS data. This decision therefore, required the use of MAP and other Added Evidence to play a less significant role than the AIMS evidence. It was determined that Added Evidence would be used as a additional measure, meaning that it would provide the state a way to give the benefit of the doubt to schools who’s data places them Achievement Profile Technical Manual Arizona Department of Education•26 near a growth cut-point. Each cell in the 6 x 6 grid represents ½ of a standard deviation (.5 SD). MAP Added Evidence data are based on the proportion of the standard deviation of a cell that the percentage of students making OYG would be allowed to affect. Utilizing the proportion of a standard deviation to set cut-points, each subject and grade is assigned Added Evidence points based on the following table: 90% + OYG .26 of a Standard Deviation 80-89% OYG 0.2 of a Standard Deviation 70-79% OYG .14 of a Standard Deviation 60-69% OYG .08 of a Standard Deviation Table 6: Added Evidence Growth Points for MAP Three Year Average % OYG Subject/Grade Mean of Growth Combination Point Distribution Math 3 Math 5 Math 8 Read 3 Read 5 Read 8 Standard Deviation 8.3 7.7 0.9 2.5 -10.1 6.2 15.02 14.08 14.33 12.15 12.33 13.95 90% + OYG 80-89% OYG 70-79% OYG 60-69% OYG .26 0.2 0.14 0.08 3.91 3.66 3.73 3.16 3.21 3.63 3.00 2.82 2.87 2.43 2.47 2.79 2.10 1.97 2.01 1.70 1.73 1.95 For example, if a school’s three year average of the percentage of students achieving OYG in math is 90%, the Added Evidence point value for MAP in grade 3 math is 3.9. These additional points are added to the AIMS change points; the possible result could be as follows: Grade 3 math: AIMS growth points + MAP Added Evidence = Total 12+ 3.9 = 15.9 Using AIMS data only, this school would not have met the .5 SD cut-point for math grade 3, which is set at 15.8 (refer to attachment 4 for the Growth cut-points for each subject grade combination). Applying the MAP Added Evidence, or the benefit of the doubt, allows this school to move beyond the .5 SD cut-point with a total point value of 15.9. Achievement Profile Technical Manual Arizona Department of Education•27 1.20 1.13 1.15 0.97 0.99 1.12 It is important to note that Added Evidence is not sufficient to move from one cut-point to another, from the .5 SD cut-point to the 1 SD cut-point, in any of the subject/grade combinations. Extended Writing In addition to using MAP for Added Evidence, a writing score is determined using trait scores on the AIMS test. It was determined that the writing portion of the AIMS assessment could serve a dual purpose: 1. Be used to both provide Added Evidence for writing, since the MAP Added Evidence only applied to math and reading 2. Provide an emphasis on the writing abilities of Arizona’s students. The AIMS Extended Writing (EW) test assesses students’ writing abilities based on six traits scores. A student receives a score from 0-6 on each of the six traits, for a score range from 0-36. The ADE determined that Added Evidence would be based on the number of students receiving a minimum total score of 24 (or an average of 4 on each trait) on the writing portion of the AIMS exam. The ADE then calculated the percentage of students in each AIMS grade for each school who received a 24 or more. This calculation was used as the basis for determining Added Evidence for the Extended Writing portion of the Achievement Profiles. Extended Writing Added Evidence points are derived from a three year average percent of students that attain a total of 24 points on the six trait scores associated with the AIMS writing test. A three year average is calculated by taking the combined three year total of the number of students that attain 24 points on the writing trait scores and dividing by the total number of students tested in each evaluated grade. (see Figure 8). Using EW in the Achievement Profile Calculation Figure 8: Calculating a 3 Year Average for EW 3 Year Average Combined three year total number of students who achieved 24 pts = X 100 Total number of students who were tested in each grade over 3 years The Added Evidence point values were determined based on the statewide distribution, by grade, of the percentage of students that achieved a total of 24 points on the AIMS Extended Writing scores. The distribution by grade is presented in Table 7. Achievement Profile Technical Manual Arizona Department of Education•28 Table 7: Distribution of Percent of Students with 24 or More Trait Score Points Mean of Distribution SD -1 SD -.5 SD Mean .5 SD 1 SD 21.7 35.6 36.4 12.18 16.43 16.84 9.52 15.61 21.7 19.17 27.385 35.6 19.56 27.98 36.4 27.79 43.815 44.82 33.88 52.03 53.24 Write 3 Write 5 Write 8 The value of the Added Evidence points granted for each grade in writing was derived directly from the statewide distribution (See Table 8). Table 8: Additional Growth Points Based on Extended Writing This percentage of students Will yield this many with 24 or more EW points EW change points From To Write 3 Write 5 Write 8 15.61 21.7 27.79 33.88 21.7 27.79 33.88 Highest 1.02 1.79 2.56 3.32 27.39 35.6 43.82 52.03 35.6 43.82 52.03 Highest 1.15 2.01 2.87 3.74 27.98 36.4 44.82 53.24 36.4 44.82 53.24 Highest 0.93 1.62 2.32 3.02 Location within state distribution From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD For example, if grade 3 in a school showed a three year average of 25.0% (the percentage of students that attained a total of 24 or more EW), the additional points for the grade 3 writing portion of the profile will be 1.79 (see Table 8). Achievement Profile Technical Manual Arizona Department of Education•29 Determining Subject/Grade Values The State Board approved the 6 x 6 grid below, which distributes individual subject/grade values based on the school’s baseline grouping as well as the school’s growth point grouping. The baseline grouping reflects the level of performance status in 2000. The possible point values are 0, 3, 5, and 7. These values were used to facilitate the setting of cut-points for each of the Achievement Profile categories by distributing the range of scores for each total subject and grade combination. Figure 9: Distribution of Subject/Grade Values 7 Baseline Grouping 6 5 0 3 3 5 5 5 4 0 3 3 5 5 5 3 0 0 3 3 5 5 2 0 0 0 3 5 5 1 0 0 1 0 2 0 3 5 4 5 5 Growth Point Grouping Distributing Subject/Grade Values The placement of the 0, 3, 5, and 7s in the 6 x 6 grid was determined using the following logic rules: 7 points A value of 7 is given if the subject/grade value meets the expectation set in state statute, 90% of students meet or exceed the standard on AIMS based on a three year average 5 Points A value of 5 is given if the subject/grade falls in: Growth group 5 or 6 or Growth group is 4 and Baseline group is greater than 3. 3 Points A value of 3 is given if the subject/grade value falls in: Growth group 2 and Baseline group is greater than 3 or Growth group 3 and Baseline group is greater than 2 or Growth group 4 and Baseline group is greater than 1 and less than 4 Achievement Profile Technical Manual Arizona Department of Education•30 6 0 Points A value of 0 is given if the subject/grade value falls in: Growth group 1 or Growth group 2 and Baseline group is less than 4 or Growth group 3 and Baseline group is less than 3 or Growth group 4 and Baseline group is less than 2 Determining an individual Subject/Grade value then requires the use of two data elements: Baseline groupings Growth point groupings A school’s Achievement Profile classification is then determined based on the evaluation of the sum of a school’s total subject/grade values. Because there are many grade configurations possible for a school the ADE set nine subject/grade combination scales for each of the possible grade combinations. Please refer to the previous section titled Subject/Grade Classification Scales for more information regarding scale setting; also see Attachment 13 for a complete list of the scales adopted by the State Board. Achievement Profile Technical Manual Arizona Department of Education•31 Graduation and Dropout Rates Overview The graduation and dropout rates are important compliments to the high school model used in the revised Achievement Profiles. Graduation rates indicate the success of students in meeting course requirements and achieving passing grades in subject areas not covered by the AIMS test. Dropout rates are included in the high school model as a measure of student persistence and to ensure that AIMS results reflect the largest percentage of the student population possible. Graduation and dropout rates are used solely in the calculation of a high school profile. This portion comprises ¼ of the total high school Achievement Profile. High school status was granted to any school that reported data in grade ten for each of the relevant school years (1999-2000, 2000-2001, and 2001-2002). In the calculation of a high school Achievement Profile, the two rates operate in conjunction with one another; in other words, the point value outcome is a reflection of how the school performs in both categories. When one of the pieces of data is missing, a school cannot meet the target for that category. Some educational facilities are required to provide an annual dropout rate, but are not required to provide a graduation rate. Under these circumstances, solely the dropout rate is used to calculate this portion of a profile (see Alternative Calculation, page 39). Rounding The rounding of dropout and graduation rate percentages worked in such a way that the decimal was only taken out to four places, regardless of the value of the fifth. In other words, the percentages, expressed at this point as decimals, were divided by 1000, then rounded based on one decimal place, such that 0.5% was rounded up and 0.4% was rounded down. Lastly, the number was divided by 10 to produce a percentage and one decimal place. For example, if, in 2000, a school reported a total enrollment of 90 and a dropout count of 25, this translates into a baseline dropout rate of 0.27777778. To round this into a percent, the calculation divided this number by 1000, which equals 277.7, then rounded to Achievement Profile Technical Manual Arizona Department of Education•32 278, then divided by 10: 27.8%. The final percent is compared to the cutoff points in each category. Graduation Rates Background The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. By examining a cohort of students who began high school at the same time, the graduation rate assesses how many students actually complete high school within a five year period (see Figure 10). Figure 10: Graduation Rate Calculation Number of Cohort members who graduated after five years Graduation = Rate Original Cohort Membership + Transfers In X 100 - Transfers - Deceased Out For more information on Arizona’s Graduation Rate methodology, please see the Graduation Rate Study published by the ADE For graduation rate data, two values are used: 1. Graduation rate for the cohort class of 2000, which represents the baseline rate and is used as a reference point in order to measure increases from year to year -and2. A two year average of the cohort classes of 2001 and 2000. The two year average is calculated by taking the total number of combined five year graduates and dividing by the total number of students in the combined cohort classes (see Figure 11). Figure 11: Calculating a Two Year Average for Graduation Rate 2 Year Average = Achievement Profile Technical Manual 2000 5 Year Graduates + 2001 5 Year Graduates Combined number of students in both cohorts X 100 Arizona Department of Education•33 Cut-points Cut-points for the graduation rate portion of the profile were determined by examining a distribution of school graduation rates in Arizona and comparing them to the state mean. After analyzing the state distribution of scores and the guidelines in the Achievement Profile legislation, the following rates were targeted as cut-point values for graduation rate: 73.5%: The state mean when school size is controlled for by averaging the rates of all schools in Arizona. 89.5%: As stated in ARS§15-241, the highest cut-point for a 5 year graduation rate is at least 90%. Meeting the Target In order for a school to meet the target for their 5 year graduation rate achievements, incremental gains must be made from one year to the next. These gains are evaluated by comparing the two year average rate to the baseline rate (see Table 9). For graduation rate data, due to a move to a five year rate, only two years of data can be used: 2000 and 2001. As with other data used in the 2002 profile, 2000 graduation rate is used as a school’s baseline. A school can meet the target for graduation rate in multiple ways, depending upon what the two year average rate is: If the two year average is 89.5% or greater, the target is automatically met. If the two year average is less than 89.5%, the baseline rate is used as a reference point and incremental gains must be made from year to year. The gains required to meet the target were derived from the statewide distribution of graduation rates. If the baseline rate is greater than or equal to 73.5%, the difference between the two year average and the baseline rate must be greater than or equal to 0.5%. If the baseline rate is less than 73.5%, the difference between the two year average and the baseline rate must be greater than or equal to 1.5%. For more information on rounding, see Rounding, page 32. Dropout Rates Background The Dropout Rate is an annual measure of how many students drop out of a school during a 12-month reporting period. It is expressed as the proportion of students who dropped out during the year to the total number of students that enrolled in the school over the course of the year (see Figure 12). The state of Arizona defines a dropout as a student who was enrolled in a school at any point during the year, was not enrolled at the end of the year and did not: Achievement Profile Technical Manual Arizona Department of Education•34 Graduate or Complete -orTransfer to another qualified educational facility -orDie Figure 12: Calculating an Annual Dropout Rate Dropout Rate = Number of students who dropped out Number of students who were enrolled during the school year X 100 For more information on Arizona’s dropout rate methodology, please see the Annual Dropout Rate Study published by the ADE. For dropout rate data, two values were used: Cut-points 1. Dropout rate for the 2000 school year, which represents the baseline rate and was used as a reference point in order to measure increases from year to year -and2. A three year average of the 2000, 2001 and 2002 school years. The three year average is calculated by taking the total number of combined dropouts and dividing by the total number of combined students served (see Figure 13). Figure 13: Calculating a Three Year Average for Dropout Rate 3 Year Average = Total number of students who dropped out in 3 years Total number of students who were enrolled during 3 years X 100 Cut-points for the drop out rate portion of the profile were determined by looking at a distribution of school dropout rates in Arizona and comparing them to the state mean. After analyzing the state distribution of scores and the guidelines in the legislation, Achievement Profile Technical Manual Arizona Department of Education•35 the following rates were targeted as cut-point values for dropout rate: 9.4%: The state mean when school size is controlled for by averaging the rates of all schools in Arizona. 6.0%: As stated in ARS§15-241, the lowest cut-point for dropout rate is less than 6% Meeting the Target In order a school meets the target for their annual dropout rate achievements, incremental decreases must be made from one year to the next. These decreases are evaluated by comparing the three year average rate to the baseline rate (see Table 9). A school can meet the target for dropout rate in multiple ways, depending upon what the three year average rate is: If the three year average for the annual dropout rate is 6.0% or less, the target is automatically met. If the three year average is greater than 6.0%, the baseline rate is used as a reference point and incremental gains must be made from year to year. The gains required to meet the target were derived from the statewide distribution of dropout rates. If the baseline rate is less than or equal to 9.4%, the difference between the three year average and the baseline rate must be greater than or equal to 0.5%. If the baseline rate is greater than 9.4%, the difference between the three year average and the baseline rate must be greater than or equal to 1.5%. For more information on rounding, see Rounding, page 32. Table 9: Target Improvements for Dropout and Graduation Rates in the High School Achievement Profile Baseline Dropout Rate* < or = 9.4% > 9.4% Target** 0.5% Decrease 1.5% Decrease Baseline Graduation Rate* > or = 73.5% < 73.5% Target** 0.5% Increase 1.5% Increase *Recall the baseline rates are taken from the 1999-2000 academic year **Meeting the target is met assessed by calculating the difference between the average rate and the baseline rate. Recall that, for the annual dropout rate, a three year average is used and for the graduation rate, a two year average is used. Achievement Profile Technical Manual Arizona Department of Education•36 Point Values and Category Outcomes In order to calculate the point outcomes for the dropout and graduation rate portion of a profile, both must be considered. The maximum number of points that can be earned is 7, and they decrease in increments to 5, 3 and 0. The combinations and associated point values are listed below. 7 Points In order to receive 7 points for the dropout and graduation rates portion of the profile, both averages must meet the highest cutpoint: The two year average for graduation must be greater than or equal to 89.5% -andThe three year average for dropout rate must be less than or equal to 6.0% Only if both of these conditions are met can a school be awarded seven points for this portion of a profile. 5 Points If both averages do not satisfy the above condition, additional criteria must be met. In order to receive five points, the targets in both dropout and graduation rates must be met. For example, assume a school’s dropout and graduation rate data have the following characteristics: Baseline dropout rate (1999-2000 school year): 7.3% 3 year average dropout rate: 6.5% Baseline graduation rate (class of 2000): 87.3% 2 year average graduation rate: 89.1% Since both averages do not meet the criteria to receive the maximum point value of 7, the average rates must be compared to the baseline rates in each category in order to evaluate gains: For graduation rate, the baseline rate is 87.3%, which is greater than 73.5%. The difference between the 2 year and the baseline rate must represent at least a 0.5% increase. In this example, the difference represents a 1.8% increase, which meets the target criteria. The dropout rate also meets the target criteria. The difference between the three year average dropout (6.5%) and the baseline (7.3%) represents a dropout decrease of 0.8%. This exceeds the target decrease of 0.5%, when the baseline rate is between 9.4% and 6.0%. Although the averages did not meet the criteria to receive seven points, both dropout and graduation rates met the targets for improvement, resulting in a point value of 5 (see Figure 14). Achievement Profile Technical Manual Arizona Department of Education•37 Figure 14: Graduation and Dropout Category Outcomes for a Point Total of 5 Dropout Rate Year Rate 2000 7.3 2001 5.7 2002 6.8 3-Year Avg 6.5 Decrease 0.8 Met Target Graduation Rate Class of Rate 2000 87.3 2001 91.2 2-Year Avg 89.1 Increase 1.8 Met Target Point Value Outcome: 5 points 3 Points In order to receive a point value of three for this portion of the profile, if both averages do not meet the criteria for a maximum point value of 7, a school must meet the target in one of the two categories. For a summary of how to meet the target in each category, see Table 10, page 40. For example, assume a school has the following data: Baseline dropout rate (1999-2000 school year): 10.1% 3 year average dropout rate: 12.2% Baseline graduation rate (class of 2000): 76.4% 2 year average graduation rate: 77.5% For dropout, the target is not met; the baseline rate is greater than 9.4% and the difference between the three year average and the baseline rates does not represent a decrease of at least 1.5%. However, the target is met for the graduation rate portion. The baseline graduation rate is greater than 73.5% and the two year average shows a 1.1% increase, which meets the target gain of at least 0.5%. This school receives a total of three points for this portion of the profile because one of the targets is met and the other is not. The report for this section will appear like the following: Achievement Profile Technical Manual Arizona Department of Education•38 Figure 15: Graduation and Dropout Category Outcomes for a Point Total of 3 Dropout Rate Year Rate 2000 10.1 2001 12.5 2002 13.2 3-Year Avg 12.2 Decrease -2.1 Graduation Rate Class of Rate 2000 76.4 2001 78.7 2-Year Avg 77.5 Increase 1.1 Met Target Did Not Meet Target Point Value Outcome: 3 points 0 Points For this portion, a school will not receive any points if neither target is met. For a summary of how to meet the target in each category, see Table 10, page 40. Assume a school has the following data: Figure 16: Graduation and Dropout Category Outcomes for a Point Total of 0 Dropout Rate Year Rate 2000 10.1 2001 12.5 2002 13.2 3-Year Avg 12.2 Decrease -2.1 Did Not Meet Target Graduation Rate Class of Rate 2000 77.6 2001 75.5 2-Year Avg 76.5 Increase -0.9 Did Not Meet Target Point Value Outcome: 0 points Alternative Calculation As mentioned, in some instances, graduation rate data is not expected from a school, but dropout rate is. Typically, those schools fall into one of the categories: The high school was open in the 1999-2000 school year but did not graduate any students. Achievement Profile Technical Manual Arizona Department of Education•39 The high school offers curriculum that leads to a traditional high school diploma, but does not offer the diploma; students must transfer to another high school in order to graduate. For high schools with one of these characteristics, only dropout rate is used in the determination of a point value for this portion of the profile. The following point outcomes will be given dependent only on the dropout rate data: For a summary of how to meet the target in each category, see Table 10, page 40. 7 Points: The maximum point value of seven is given if the three year average for dropout is less than or equal to 6.0%. 5 Points: If the three year average is greater than 6.0%, the difference between the three year average and the baseline rate must meet the appropriate criteria, as listed in Table 1. 0 Points: If the three year average is greater than 6.0%, and the target difference between the three year average and the baseline rate is not met, the school will not be awarded any points for this portion of its profile. Summary In conclusion, the dropout and graduation rates, when both are expected from a school, work together in order to produce a point value outcome for this portion of a school’s profile. The following table summarizes these outcomes: Table 10: Point Value Outcomes For Dropout and Graduation Rates If the 3 Year Dropout Rate average is < or = 6.0% AND the 2 year graduation rate average is: > or = 89.5% The Point Value Outcome is: 7 Dropout Rate: Met Target? Yes No Yes No Graduation Rate: Met Target? Yes Yes No No Point Value Outcome 5 3 3 0 If only the dropout rate is expected from a school, the following outcomes are possible: Achievement Profile Technical Manual Arizona Department of Education•40 Table 11: Point Value Outcomes for Dropout Rate: Alternative Calculation If the 3 Year Dropout Rate The Point Value Outcome is: average is < or = 6.0% 7 Dropout Rate: Met Target? Yes No Achievement Profile Technical Manual Point Value Outcome 5 0 Arizona Department of Education•41 Small School Score Adjustment Criteria Some schools are eligible for low score outlier removal as a result of small overall size. The following criteria are used to determine which schools should receive this adjustment: The school has an average daily membership (ADM) of ninetynine students or fewer and the school has AIMS scores for ninetynine students or fewer. Or, if the school did not report ADM but has valid AIMS scores for ninety-nine students or fewer. Method Small school adjustments were made to schools that met these criteria. The following steps were taken in adjusting the scores of qualified schools: 1. Low and high score outlier identification was conducted in each of the subjects of reading, writing and math for grades 3, 5, 8 and high school respectively. A decision was reached to remove low scores only. It was reasoned that statistically low outlier scores represented chance occurrences of poor individual level test performance. Removing low outlier scores from a distribution of test scores was particularly important for schools with small size because it negated the impact that a very low score(s) might have on the overall picture of the school’s performance. These scores were identified by employing a bivariate regression model where each school’s mean scale score was regressed on a two-category indicator for the school: Where 1 = the student was tested in that school, and 0 = the student was not tested in the school. 2. A unique equation was estimated for each school in order to produce a statistic that would address the question about the affect that any one student’s score within that school Achievement Profile Technical Manual Arizona Department of Education•42 would have on the small school’s overall mean scale score. This equation took the following form: Mean scale scorei = α + b School Dummyi In this application, the regression coefficient plus or minus the intercept represented the school’s mean scale score. 3. The DFFIT statistic was computed for each score within the school and saved to a separate data file for later analysis. The DFFIT statistic assesses the change that a particular case will have on the predicted value i.e., the small school’s mean scale score when it has been deleted from the regression equation (Belsey et al. 1980)2. The following equation was used: DFFITSi = ti hi 1 − hi where ti = studentized residual and hi = leverage value 4. Once the DFFITS were estimated for each case the standard deviation of the statistic was computed within each school. A cut-point was set such that any student with a DFFIT score greater or less than two standard deviations above or below the within school mean DFFIT could be considered an outlier relative to the distribution of test scores of that school. 5. Students were assigned an identification number so that their observation could be flagged if it fell above or below this school specific cut-point. Scores that had been identified statistically as outliers and were at the low end of the school’s test score distribution were eliminated from the datasets that were used to compute the final performance label. 2 Belsley, D.A., Kuh, E. and Welsch, R.E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: John Wiley. Achievement Profile Technical Manual Arizona Department of Education•43 Exempted Schools Overview For the 2002-2003 Achievement Profiles, certain types of schools were exempted from receiving a profile. Due to a wide variety of schools with vastly different characteristics in Arizona, modified formulae will be created in order to evaluate them appropriately. There are five distinct categories of schools that did not receive profiles in the 2002-2003 school year: Extremely Small Schools Alternative Schools Accommodation Schools New Schools K-2 Schools Extremely Small Schools The criteria used to classify schools as extremely small in size for the purpose of computing an Achievement Profile via Arizona’s accountability system differed from the process used to eliminate low test score outliers. Although all schools classified as small in the “low score adjustment” process were eligible for low score removal, there was a subset of schools where the Average Daily Membership (ADM) and the number of students tested were too small to be classified using the standard methodology. Extremely small schools cannot be evaluated using the existing model of performance categories and change scores because of the small number of students tested. Given the architecture of the present accountability system there is a direct relationship between the number of students tested in a school and the overall reliability of the school Achievement Profile. In some cases, particularly where the number of students tested was less than or equal to 50, the movement of one, two or three students from one performance category to another directly affected the number of change points a school was awarded. Due to the fact that the smallest schools in the state were most likely to be effected by the conventional computation of their Achievement Profile, the State Board of Education determined that extremely small schools would not receive an Achievement Profile. A discussion of the criteria that were used to determine which schools were small enough to be excluded follows. Achievement Profile Technical Manual Arizona Department of Education•44 A series of steps were taken to determine which small schools could be classified as extremely small. 1. First, a school was examined to determine how many grade/subject combinations it had for each test year. For example, grade school X may have had the following grade/subject combinations for each AIMS test year. Table 12: Grade School Xa Grade 3 Grade 5 Grade 8 ADM AIMS test year 99-00 Math (7) Reading (16) Writing (17) Math (10) Reading (18) Writing (16) Math (6) Reading (16) Writing (16) 99 AIMS test year 00-01b Math (8) Reading (10) Writing (6) Math (9) Reading (13) Writing (8) Math (5) Reading (10) Writing (6) 80 AIMS test year 01-02 Math (7) Reading (12) Writing (5) Math (4) Reading (10) Writing (7) Math (4) Reading (12) Writing (7) 75 a The number of students tested are shown in parentheses. Average daily membership (ADM) for the school is shown in the last row. ADM and the number of students tested in the school were rarely equivalent. b The AIMS scores for the math test given during the 1999-2000 school year proved to be incompatible with the 2000-2001 and 2001-2002 versions of the high school math test. Hence, these scores could not be used. 2. School data was accessed to determine if the school had an ADM of less than 100 students enrolled in any of the three tests years. If the school met this minimum condition the school data was referenced again to see if 15 students or less were tested in the 1999-2000 baseline test year. This process was iteratively executed for each grade and each subject within the specific grade. If both criteria were met in the grade and subject area the school received an “S” (for very small) in the respective cell. Given the numbers in the table shown above, the conversion of Table 11, would appear as Table 12. Achievement Profile Technical Manual Arizona Department of Education•45 Table 13: Grade School X Grade 3 Grade 5 Grade 8 ADM AIMS test year 99-00 Math (S) Reading (16) Writing (17) Math (S) Reading (18) Writing (16) Math (S) Reading (16) Writing (16) 99 AIMS test year 00-01 Math (8) Reading (10) Writing (6) Math (9) Reading (13) Writing (8) Math (5) Reading (10) Writing (6) 80 AIMS test year 01-02 Math (7) Reading (12) Writing (5) Math (4) Reading (10) Writing (7) Math (4) Reading (12) Writing (7) 75 3. The final step to determine if the school would be considered extremely small was based on similar results shown in Table 12. It was decided that if over 1/3 or 33% of all the grade/subject combinations were identified as “S” then the whole school would be considered too small to accurately compute an Achievement Profile. This decision was made based on the three (3) subject/grade value scale which resulted in allowing one of the three values to be identified as “S” and still produce an Achievement Profile, while two of the three values being identified as “S” would not produce a profile. This logic was then carried out into each of the other subject/grade value scales and ultimately produced the greater than 1/3 rule for an “extremely small” school classification. Such schools meeting this requirement would be identified as extremely small. According to this criterion School X would not have been considered too small to receive an Achievement Profile because only 3 of 9 grade/subject combinations received a “S” - 3/9 = .33 or 33%. Alternative Schools A public school desiring identification as an alternative school must apply to the Arizona State Board of Education for such status. These schools must be separate entities according to Arizona school finance provisions (funded as a school, reported as a school, etc.). Alternative school status was not granted to a program within a school. Following the passage of ARS§15-241 the Arizona Department of Education (ADE) established a committee of educators with expertise in the field of alternative education to develop the elemental criteria used to define an alternative school. The ADE took great care in soliciting the participation of educators representing a wide variety of schools, Local Education Agencies Achievement Profile Technical Manual Arizona Department of Education•46 (LEAs), and communities. Based on the committee’s recommendations, an alternative school is a school that the Arizona State Board of Education determined met all of the following criteria: 1. A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion #3) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. Note: The school’s mission statement or charter must be communicated to the public. 2. The educational program and related student services of the school must match the mission or charter of the school. 3. The school must intend to serve students exclusively in one or more of the following categories: Students with behavioral issues (documented history of disruptive behavior) Students identified as dropouts Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades Pregnant and/or parenting students Adjudicated youth 4. Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Also, no public school district may have more than ten percent (10%) of their total student population attending an alternative school or any combination of alternative schools served by the district at one time. Smaller districts, if they wish, may participate in the development of a “consortium” alternative school. In future years, alternative schools will receive profiles based on a modified formula, as a result of the unique student composition attending these schools. The formula will have the following characteristics: Ninety-five percent (95%) of students enrolled must take Arizona’s Instrument to Measure Standards (AIMS). Criteria: The ADE will develop a consistent formula to determine the percentage of students taking AIMS for all Achievement Profile Technical Manual Arizona Department of Education•47 public schools and will apply this formula to alternative schools. Decrease Dropout rate. Criteria: Alternative schools will have the same Annual Dropout Rate targets as conventional public schools (see Table 10, page 40). Increase the percentage of graduates who demonstrate proficiency on the Standards via AIMS. Criteria: The 2002 academic year will serve as the baseline. Every alternative school is expected to have 100% of graduates demonstrate proficiency of the Standards via AIMS by 2006. The expected annual progress for each alternative school is calculated as follows: Subtract the percentage of graduates who also demonstrate proficiency of the Standards on the 2000 AIMS from 100%. Divide the remainder by four (4). Alternative School Application Process In order to receive alternative school status during the 2002-2003 academic year, each school desiring such status had to complete the application process. The electronic application was accessed through the ADE’s common log-on site allowing each public elementary and secondary school in Arizona the opportunity to apply for alternative school status. The application for alternative school status consisted of the following four (4) questions: 1.) What is the stated mission of the school? Does the mission specifically state that the school will serve students in one or more of the subgroups defined by the Arizona State Board of Education? 2.) How has the mission of the school been communicated to the public? 3.) How is the educational program (e.g. curriculum, instruction, supplemental services, etc.) tailored to meet the needs of students in one or more subgroups defined by the Arizona State Board of Education? 4.) [Charter Schools Only] Does the school’s charter specifically state the school will serve students in one or more of the subgroups identified by the Arizona State Board of Education. Schools completing the application for alternative school status resubmitted their responses through the common log-on site. A committee comprised of one representative from the ADE, one staff representative from the Arizona State Board of Education, and one staff representative from the Arizona State Charter Board then reviewed each application. In order to recommend approval of alternative school status by the Arizona State Board of Education, Achievement Profile Technical Manual Arizona Department of Education•48 the three-member committee had to reach a consensus regarding each applicant’s intent to meet the criteria established by the Arizona State Board of Education. The three-member committee submitted a list of schools recommended for alternative school status to the Arizona State Board of Education for approval during a public meeting held on October 28, 2002. While approving the list of schools recommended for alternative school status, the Arizona State Board of Education also approved a measure providing an appeals process for schools not receiving a recommendation of alternative school status from the review committee. In order to appeal the recommendation of the review committee, schools were given seven working days to submit additional evidence proving intent to meet the criteria established by the Arizona State Board of Education. The three members of the review committee then reconvened to evaluate the additional evidence submitted by each school appealing the committee’s initial decision. Once again, the review committee had to reach a consensus regarding the each appellant’s intent to meet the criteria of an alternative school as indicated by additional evidence submitted to the committee. The review committee then submitted a list of appellant schools recommended for alternative school status to the Arizona State Board of Education for approval during a public meeting held on November 18, 2002. Accommodation Schools The ADE met with the County Superintendents’ offices in order to determine the policy for calculating Achievement Profiles for Accommodation Schools. These schools are, by statue, regulated and maintained by the County Superintendents. The ADE along with this governing body determined that Accommodation schools could be placed into three distinct categories: Similar to Traditional Schools Alternative Schools and Detention Centers Accommodation schools that could be identified as Similar to Traditional schools were given an Achievement Profile based on the standard methodology outlined for public schools. Schools that could be identified for Alternative School Status were granted this status and will receive an Achievement Profile as outlined in the methodology for Alternative schools. Lastly, schools identified as Detention Centers were determined to not require an Achievement Profile, therefore these particular Accommodation schools will not have an Achievement Profile calculated at this time. New Schools A certain percentage of schools currently in operation were not in operation for three consecutive academic years in order to provide all data necessary to calculate an Achievement Profile. Schools that do not have three years of data due to their recent opening will not receive a profile until they are in operation for three Achievement Profile Technical Manual Arizona Department of Education•49 consecutive years. Their initial year of operation will serve as the baseline year for these schools. For example, if a school’s first year in operation was the 20012002 academic year, the data reported for this year will function as the baseline data for that school. A profile will not be calculated until it is in operation for three consecutive years, for example, in the fall of 2004. K-2 Schools Schools exclusively serving the kindergarten to second grade student populations will not receive an Achievement Profile at this time, but will in future years. These grades are not currently assessed via our state assessment (AIMS) and the ADE cannot compute a measure of academic progress (MAP) for these schools, both of which are key components to the elementary model. Achievement Profile Technical Manual Arizona Department of Education•50 Putting it All Together: How to Calculate an Achievement Profile Calculating an Elementary School Achievement Profile In order to accurately compute an Achievement Profile for a school, several pieces of data must be collected. Elementary Schools must have the following data: Three years of AIMS scores in reading, writing and math (2000, 2001 and 2002) MAP (2000, 2001 and 2002). These data are used to provide Added Evidence for a school’s Achievement Profile EWS (2000, 2001 and 2002). Just as MAP, the Extended Writing Scores are used to provide Added Evidence. High schools must have: Three years of reading and writing AIMS data (2000, 2001 and 2002) and two years of math (2001 and 2002) Three years of dropout rate (2000, 2001 and 2002) 2 years of graduation rate (2000 and 2001). Some schools may not require graduation rate data (see Alternative Calculation, page 39) Step 1: Determine Baseline Grouping A school’s baseline grouping is determined from its 2000 AIMS data (baseline data). In each grade and subject area, the percent of students that met or exceeded the standard is computed and compared to the appropriate state groupings, as determined by the federal NCLB legislation. For example, if, in 2000, 84% of the students in grade 3 math met or exceeded the standard, the baseline grouping for that area would be 5. Step 2: Determine Growth Points for AIMS Determining a school’s change points for each subject/grade combination on the AIMS portion of a profile is based on student movement out of the FFB category and student movement into the Meet/EXC category, according to the difference between the three Achievement Profile Technical Manual Arizona Department of Education•51 year average and the baseline percentages. A three year average is calculated by adding the total number of students in each category over three years and dividing by the total number of students tested for each subject/grade combination (see Figure 17). Figure 17: Calculating a 3 Year Average for %FFB and Meet/Exc Total Number of Students that FFB or Meet/EXC = 3 Year Ave X 100 Total Number of Students Tested Over 3 Years High school AIMS math test does not use the academic year 2000 for the baseline year. The baseline for this subject/grade combination is 2001. It is considered a positive change if the three year average percentage of students in the Meet/EXC category is higher than the baseline percentage, or the three year average of students in the FFB is lower than the baseline percentage. It is considered a negative change if the three year average percentage of students in the Meet/EXC category is lower than the baseline, or if the three year average percent of students in the FFB category is higher than the baseline percentage. It is expected that schools will increase the percentage of students that meet the standards over time, not increase the percentage that falls far below the standard (see Figure 18). Figure 18: Positive Growth Points FFB Achievement Profile Technical Manual APP MEET/EXC Arizona Department of Education•52 For example, assume a school has the following data: Grade 3 Subject Math Academic Year 2000 2001 2002 % FFB 25.56 37.80 28.57 % Meet/Exc 34.44 32.93 35.71 3 Year Averages 30.70 34.21 The three year average for the percentage of students in the FFB category is higher than the baseline percentage, which is a negative change. Likewise, the three year average for students that met the standard is slightly lower than the baseline percentage (34.21 compared to a baseline of 34.44). This is also a negative change; a school should increase the percentage of students meeting the state standard over time. The change points associated with grade 3 math for this school will be determined according to the following: FFB Meet/EXC 3 Year Average 30.7 34.21 Baseline 25.56 34.44 Difference -5.14 -0.23 Since these changes both represent a negative change, in terms of the academic growth in this example, the values will be negative, such that the total change points in grade 3 math for this school will be -5.37. If the difference between the three year average and the baseline in the Meet/EXC category were positive, the total change points would have reflected this positive change: -5.14 +0.23 = -4.91. Step 3: Determine Growth Points for Added Evidence The Stanford 9 test is distributed to elementary grades two through nine and measures student performance in the areas of math and reading. Additionally, unlike the AIMS growth points, Added Evidence points are determined by the performance of the entire school. Math and reading Added Evidence scores are taken from a three year average of the percentage of students in each school, and in each subject, that accomplish OYG from one year to the next. A three year average is calculated by taking the combined total number of students that make OYG in math and reading and dividing by the total number of students tested in math and reading (see Figure 19). Achievement Profile Technical Manual Arizona Department of Education•53 Figure 19: Calculating a 3 Year Average for MAP 3 Year Combined three year total number of students who made OYG (Math and Reading) X 100 Average = Total number of students who were tested over 3 years After a school level three year average percent of students making OYG in math and reading is calculated for each school, growth points may be attributed to each grade and subject area if the percentage is 60% or greater. Table 14: Added Evidence Growth Points for MAP Three Year Average % OYG Subject/Grade Mean of Growth Combination Point Distribution Math 3 Math 5 Math 8 Read 3 Read 5 Read 8 8.3 7.7 0.9 2.5 -10.1 6.2 Standard Deviation 15.02 14.08 14.33 12.15 12.33 13.95 90% + OYG 80-89% OYG 70-79% OYG 60-69% OYG .26 SD 0.2 0.14 0.08 3.91 3.66 3.73 3.16 3.21 3.63 3.00 2.82 2.87 2.43 2.47 2.79 2.10 1.97 2.01 1.70 1.73 1.95 For example, if a school’s three year average of the percentage of students achieving OYG in math is 90%, the Added Evidence point value for MAP in grade 3 math is 3.9. Extended Writing Added Evidence points are derived from a three year average percent of students that attain a total of 24 points on the six trait scores associated with the AIMS writing test. A three year average is calculated by taking the combined three year total of the number of students that attain 24 points on the writing trait scores and dividing by the total number of students tested in each evaluated grade. (see Figure 20). Achievement Profile Technical Manual Arizona Department of Education•54 1.20 1.13 1.15 0.97 0.99 1.12 Figure 20: Calculating a 3 Year Average for EW 3 Year Average Combined three year total number of students who achieved 24 pts = X 100 Total number of students who were tested in each grade over 3 years The value of the Added Evidence points granted for each grade in writing was derived directly from the statewide distribution (See Table 15). Table 15: Additional Growth Points Based on Extended Writing This percentage of students Will yield this many with 24 or more EW points EW change points From To Write 3 Write 5 Write 8 15.61 21.7 27.79 33.88 21.7 27.79 33.88 Highest 1.02 1.79 2.56 3.32 27.39 35.6 43.82 52.03 35.6 43.82 52.03 Highest 1.15 2.01 2.87 3.74 27.98 36.4 44.82 53.24 36.4 44.82 53.24 Highest 0.93 1.62 2.32 3.02 Location within state distribution From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD From -.5 SD to Mean Mean to .5 SD .5 to 1 SD Greater than 1 SD For example, if grade 3 in a school showed a three year average of 25.0% (the percentage of students that attained a total of 24 or more EW), the additional points for the grade 3 writing portion of the profile will be 1.79 (see Table 15). Step 4: Determine Total Growth Points Total growth points are calculated for each subject/grade combination in a school by adding together the AIMS change points and the Added Evidence points. For example, if a school had 78% make OYG in math, the change points for MAP in grade 3 math would be 2.10. For more information, see Using MAP in the Achievement Profile Calculation, page 26. Achievement Profile Technical Manual Arizona Department of Education•55 Also, suppose the AIMS change points in the same subject and grade were 8.36. For more information, see Determining Change Points for a School’s Subject/Grade Combination, page 21. These points are added together to determine a school’s total growth points for grade 3 math: Figure 21: Calculating a School’s Total Change Points for Grade 3 Math AIMS + MAP = Total Change Points 8.36 +2.10 = 10.46 Step 5: Determine the Appropriate Point Outcome Grid After total growth points are calculated for each subject/grade combination, the appropriate 6x6 grid is used to determine the point outcomes for that portion. Each grid is unique to a subject/grade combination and is derived from the baseline groupings and the statewide distribution of change points. Baseline Grouping Table 16: Subject/Grade Values Grid 6 5 4 3 2 0 0 0 0 1 0 7 3 3 0 0 0 1 3 3 3 0 5 5 3 3 0 0 2 3 Growth Point Grouping 5 5 5 5 5 5 5 5 5 4 5 5 Vertical Axis: Baseline groupings, which are calculated by following the NCLB methodology for determining the state’s “starting point” (see Baseline Groupings: Y Axis, page 18). Horizontal Axis: Growth point groupings, based on the state distribution for each subject/grade combination (see Change Point Groupings: X Axis, page 20). For example, for grade 3 math, the point outcomes and axis values will look like the following: Achievement Profile Technical Manual Arizona Department of Education•56 6 Baseline Groupings Figure 22: Example: Math Grade 3 Baseline Grouping and Growth Cut Points 7 90 0 0 0 0 0 75 61 48 32 3 3 0 0 0 5 5 3 3 0 3 3 3 0 0 5 5 5 5 5 5 5 5 5 5 -6.72 .79 8.3 15.81 23.32 -1.0 SD -.5 SD X .5 SD 1.0 SD Growth Point Groupings Notice that a maximum of 7 points is given to a combination in which the 3 year average percent of students that has met or exceeded the standard on AIMS is 90% or greater, regardless of the number of change points. For more information on AIMS performance categories, such as MEET or EXC, see Arizona Instrument to Measure Standards, page 18). In the example above, since this school’s 3 year average percent of students meeting or exceeding the standard is less than 90%, the baseline figure is used. The baseline percent for AIMS is 84% MEET or EXC and the total change points for grade 3 math is 10.46. Therefore, the point value for this portion of the profile would be, according to the grid in Figure 6, equal to 5. This process is repeated for each subject and grade combination in a school. Step 6: Determine Total Point Outcomes All point outcomes are added together and compared to a scale range in order to determine a final Achievement Profile for the school. The scales are varied based on the number of grade and test combinations in a school; if a school only serves grade 3, the scale is based on three categories (reading, writing and math) and if a school has grades 3 and 5, the label ranges are based on six categories (three subjects for each grade). Suppose an elementary school serves grades kindergarten through 8, with the following point values based on AIMS and Added Evidence: Figure 23: Point Value Outcomes for a K-8 School Achievement Profile Technical Manual Grade Reading Writing Math 3 5 7 5 17 5 5 5 5 15 8 0 0 5 5 TOTAL 37 Arizona Department of Education•57 Step 7: Determine Subject/Grade Value Scale After adding all subject/grade values together to achieve a total of 37, this value is compared to the nine subject/grade value scale, as this school has nine subject/grade combinations. Figure 24: Nine Subject/Grade Values Scale SCALE Underperforming Maintaining Improving Excelling 0-14 15-35 36-62 63-63 In this example, the school will receive an Achievement Profile of Improving because the total point outcome is 37. For a complete look at the different label ranges, please see Appendix 9, page 75. Calculating a High School Profile A high school profile is calculated using AIMS change points, graduation and dropout rates; Added Evidence points are not included. In the end, points from the AIMS portion are added to the total points gained from the dropout and graduation rate portion. Step 1: Determine Baseline Grouping A school’s baseline grouping is determined from its 2000 AIMS reading and writing data and 2001 math data (baseline data). In each grade and subject area, the percent of students that met or exceeded the standard is computed and compared to the appropriate state groupings, as determined by the federal NCLB legislation. For example, if, in 2000, 84% of the students in high school writing met or exceeded the standard, the baseline grouping for that area would be. Step 2: Determine Growth Points for AIMS Determining a school’s change points for each subject/grade combination on the AIMS portion of a profile is based on student movement out of the FFB category and student movement into the Meet/EXC category, according to the difference between the three year average and the baseline percentages. A three year average is calculated by adding the total number of students in each category over three years and dividing by the total number of students tested for each subject/grade combination (see Figure 25). Achievement Profile Technical Manual Arizona Department of Education•58 Figure 25: Calculating a 3 Year Average for %FFB and Meet/Exc = 3 Year Ave Total Number of Students that FFB or Meet/EXC X 100 Total Number of Students Tested Over 3 Years High school AIMS math test does not use the academic year 2000 for the baseline year. The baseline for this subject/grade combination is 2001. It is considered a positive change if the three year average percentage of students in the Meet/EXC category is higher than the baseline percentage, or the three year average of students in the FFB is lower than the baseline percentage. It is considered a negative change if the three year average percentage of students in the Meet/EXC category is lower than the baseline, or if the three year average percent of students in the FFB category is higher than the baseline percentage. It is expected that schools will increase the percentage of students that meet the standards over time, not increase the percentage that falls far below the standard (see Figure 26). Figure 26: Positive Growth Points FFB APP MEET/EXC For example, assume a school has the following data: Grade High Subject Writing Academic Year 2000 2001 2002 % FFB 25.56 37.80 28.57 % Meet/Exc 34.44 32.93 35.71 3 Year Averages 30.70 34.21 The three year average for the percentage of students in the FFB category is higher than the baseline percentage, which is a negative change. Likewise, the three year average for students that met the standard is slightly lower than the baseline percentage (34.21 compared to a baseline of 34.44). This is also a negative change; a Achievement Profile Technical Manual Arizona Department of Education•59 school should increase the percentage of students meeting the state standard over time. The change points associated with grade high school writing for this school will be determined according to the following: 3 Year Average 30.7 34.21 FFB Meet/EXC Baseline 25.56 34.44 Difference -5.14 -0.23 Since these changes both represent a negative change, in terms of the academic growth in this example, the values will be negative, such that the total change points in high school writing for this school will be -5.37. If the difference between the three year average and the baseline in the Meet/EXC category were positive, the total change points would have reflected this positive change: -5.14 +0.23 = -4.91. Step 3: Determine the Appropriate Point Outcome Grid After total growth points are calculated for each subject/grade combination, the appropriate 6x6 grid is used to determine the point outcomes for that portion. Each grid is unique to a subject/grade combination and is derived from the baseline groupings and the statewide distribution of change points. Baseline Grouping Table 17: Subject/Grade Values Grid 6 5 4 3 2 0 0 0 0 1 0 7 3 3 0 0 0 1 3 3 3 0 5 5 3 3 0 0 2 3 4 Growth Point Grouping 5 5 5 5 5 5 5 5 5 5 5 Vertical Axis: Baseline groupings which are calculated by following the NCLB methodology for determining the state’s “starting point” (see Baseline Groupings: Y Axis, page 18). Horizontal Axis: Growth point groupings, based on the state distribution for each subject/grade combination (see Change Point Groupings: X Axis, page 20). For example, for high school writing, the point outcomes and axis values will look like the following: Achievement Profile Technical Manual Arizona Department of Education•60 6 Figure 27: Example: Math Grade 10 Baseline Grouping and Growth Cut Points 7 Baseline Groupings 90 3 3 0 0 0 0 0 0 0 0 80 70 60 45 3 3 3 0 0 5 5 3 3 0 5 5 5 5 5 5 5 5 5 5 -8.19 -3.95 0.3 4.55 8.79 -1.0 SD -.5 SD X .5 SD 1.0 SD Growth Point Groupings Notice that a maximum of 7 points is given to a combination in which the 3 year average percent of students that has met or exceeded the standard on AIMS is 90% or greater, regardless of the number of change points. For more information on AIMS performance categories, such as MEET or EXC, see Arizona Instrument to Measure Standards, page 18). In the example above, since this school’s 3 year average percent of students meeting or exceeding the standard is less than 90%, the baseline figure is used. The baseline percent for AIMS is 84% MEET or EXC and the total change points for high school writing is –5.37. Therefore, the point value for this portion of the profile would be, according to the grid in Figure 11, equal to 3. This process is repeated for each subject and grade combination in a school. Step 4: Determine the Outcome Values for Dropout and Graduation Rate In order for a school to meet the target for their annual dropout rate achievements, incremental decreases must be made from one year to the next. These decreases are evaluated by comparing the three year average rate to the baseline rate. A school can meet the target for dropout rate in multiple ways, depending upon what the three year average rate is: If the three year average for the annual dropout rate is 6.0% or less, the target is automatically met. If the three year average is greater than 6.0%, the baseline rate is used as a reference point. If the baseline rate is less than or equal to 9.4%, the difference between the three year average and the baseline rate must be greater than or equal to 0.5%. Achievement Profile Technical Manual Arizona Department of Education•61 If the baseline rate is greater than 9.4%, the difference between the three year average and the baseline rate must be greater than or equal to 1.5%. For more information on rounding, see Rounding, page 32. In order a school meets the target for their 5 year graduation rate achievements, incremental gains must be made from one year to the next. These gains are evaluated by comparing the two year average rate to the baseline rate. For graduation rate data, due to a move to a five year rate, only two years of data can be used: 2000 and 2001. As with all other data used in the 2002 profile, 2000 graduation rate is used as a school’s baseline. A school can meet the target for graduation rate in multiple ways, depending upon what the two year average rate is: If the two year average is 89.5% or greater, the target is automatically met. If the two year average is less than 89.5%, the baseline rate is used as a reference point. If the baseline rate is greater than or equal to 73.5%, the difference between the two year average and the baseline rate must be greater than or equal to 0.5%. If the baseline rate is less than 73.5%, the difference between the two year average and the baseline rate must be greater than or equal to 1.5%. The dropout and graduation rates, when both are expected from a school, work together in order to produce a point value outcome for this portion of a school’s profile. The following table summarizes these outcomes: Table 18: Point Value Outcomes For Dropout and Graduation Rates If the 3 Year Dropout Rate average is < or = 6.0% AND the 2 year graduation rate average is: > or = 89.5% The Point Value Outcome is: 7 Dropout Rate: Met Target? Yes No Yes No Graduation Rate: Met Target? Yes Yes No No Point Value Outcome 5 3 3 0 Achievement Profile Technical Manual Arizona Department of Education•62 If only the dropout rate is expected from a school, the following outcomes are possible: Table 19: Point Value Outcomes for Dropout Rate: Alternative Calculation If the 3 Year Dropout Rate The Point Value Outcome is: average is < or = 6.0% 7 Dropout Rate: Met Target? Yes No Step 5: Determine Total Point Outcomes Point Value Outcome 5 0 All point outcomes are added together and compared to a scale range in order to determine a final Achievement Profile for the school. Suppose a high school was determined to have the following point value outcomes for AIMS and dropout and graduation rate: Figure 28: High School Four Subject/Grade Values Grade Reading Writing Math Grad/DO HS 3 3 3 0 9 TOTAL 9 The total points are then compared to the label ranges for the high school model: Figure 29: Four Subject/Grade Value Scale Scale Underperforming Maintaining Improving Excelling 0-5 6-14 15-27 28-28 This school would receive a Maintaining Achievement Profile based on the high school label ranges. For a complete look at the different label ranges, please see Appendix 9, page 75. Achievement Profile Technical Manual Arizona Department of Education•63 List of Appendices APPENDIX 1: BASELINE GROUPING SEPARATION POINTS CALCULATED ON 2000 DATA WITH NCLB METHODOLOGY .......................................................................................................... 65 APPENDIX 1 (CONT’D):............................................................... 66 APPENDIX 2: ADDED EVIDENCE: GROWTH POINTS FOR MAP................................................................................................. 67 APPENDIX 3: ADDITIONAL GROWTH POINTS BASED ON EXTENDED WRITING (EW) ........................................................ 68 APPENDIX 4: GROWTH CUT-POINTS FOR EACH SUBJECT/GRADE COMBINATION ............................................ 69 APPENDIX 5: GRADUATION AND DROPOUT RATES TARGETS AND VALUES ............................................................. 70 APPENDIX 6: SUBJECT GRADE/LEVEL TABLE ..................... 71 APPENDIX 7: ALTERNATIVE SCHOOL DEFINITION ............ 72 APPENDIX 7 (CONT’D):............................................................... 73 APPENDIX 8: SMALL SCHOOLS ADJUSTMENT..................... 74 APPENDIX 9: SUBJECT/GRADE VALUE CLASSIFICATION SCALES-ALL GRADES................................................................. 75 APPENDIX 9 (CONT’D):............................................................... 76 APPENDIX 9 (CONT’D):............................................................... 77 APPENDIX 10: STATE BOARD INFORMATION PACKET...... 78 Achievement Profile Technical Manual Arizona Department of Education•64 Appendix 1: Baseline Grouping Separation Points Calculated on 2000 Data with NCLB Methodology Reading Math Writing Cut Cut Cut Group Point Group Point Group Point Grade 3 Group 6* 90-100 Group 5 88-89 Group 4 79-87 Group 3 69-78 Group 2 54-68 Group 1 0-53 90-100 88 79 69 54 90-100 75 75-89 93 61 61-74 88-92 48 48-60 80-87 32 32-47 64-79 Maintaining Line 0-31 0-63 90-100 90-100 93 88 80 64 Grade 5 Group 6* 90-100 84 74 62 45 52 52-89 73-89 40 40-51 61-72 27 27-39 50-60 14 14-26 34-49 Maintaining Line Group 5 84-89 Group 4 74-83 Group 3 62-73 Group 2 45-61 Group 1 0-44 0-13 0-33 Group 6* 90-100 90-100 90-100 Group 5 70-89 Group 4 59-69 Group 3 48-58 Group 2 34-47 Group 1 0-33 73 61 50 34 Grade 8 Achievement Profile Technical Manual 70 59 48 34 30 67-89 20 20-29 55-66 12 12-19 43-54 6 6-11 30-42 Maintaining Line 30-89 0-5 67 55 43 30 0-29 Arizona Department of Education•65 Appendix 1 (Cont’d): High School Group 6* 90-100 Group 5 80-89 Group 4 70-79 Group 3 60-69 Group 2 45-59 Group 1 0-44 90-100 80 70 60 45 90-100 41 44-89 31 31-40 33-43 22 21-30 24-32 11 11-21 14-23 Maintaining Line 41-89 0-10 44 33 24 14 0-13 Note: All cells represent percentage of students Meeting or Exceeding the Standards on 2000 AIMS *Group 6 includes schools with a three-year average of at least 90% of students Meeting or Exceeding the Standards on AIMS Achievement Profile Technical Manual Arizona Department of Education•66 Appendix 2: Added Evidence: Growth Points for MAP %Students Making One Year's Growth (OYG) 90% 80-89% 70-79% 60-69% Subject/Grade SD 0.26 0.2 0.14 0.08 Math 3 5 8 15.02 14.08 14.33 3.91 3.66 3.73 3.00 2.82 2.87 2.10 1.97 2.01 1.20 1.13 1.15 Reading 3 5 8 12.15 12.33 13.95 3.16 3.21 3.63 2.43 2.47 2.79 1.70 1.73 1.95 0.97 0.99 1.12 Achievement Profile Technical Manual Arizona Department of Education•67 Appendix 3: Additional Growth Points Based on Extended Writing (EW) Percentage of Students with 24 or more EW Points Will Yield this many From 15.6 21.7 27.8 33.9 27.4 35.6 43.8 52 28 36.4 44.8 53.2 EW Change Points 1.02 1.79 2.56 3.32 1.15 2.01 2.87 3.74 0.93 1.62 2.32 3.02 Grade 3 Grade 5 Grade 8 Achievement Profile Technical Manual To 21.7 27.8 33.9 Highest 35.6 43.8 52 Highest 36.4 44.8 53.2 Highest Arizona Department of Education•68 Appendix 4: Growth Cut-Points for Each Subject/Grade Combination Cut Points Subject/Grade State Mean SD -1 SD -.5 SD Mean .5 SD 1 SD Math 3 8.3 15.02 -6.72 0.79 8.3 15.81 23.32 5 7.7 14.08 -6.38 0.66 7.7 14.74 21.78 8 0.9 14.33 -13.43 -6.27 0.9 8.07 15.23 3 2.5 12.15 -9.65 -3.58 2.5 8.58 14.65 5 -10.1 8 6.2 13.95 -7.75 -0.78 6.2 13.18 20.15 3 -3.1 12.78 -15.88 -9.49 -3.1 3.29 5 0.8 14.37 -13.57 -6.39 0.8 7.99 15.17 8 -3.6 11.6 -15.2 -9.4 -3.6 2.2 8 Math 0.3 8.49 -8.19 -3.95 0.3 4.55 8.79 Reading 1.1 15.51 -14.41 -6.66 1.1 8.86 16.61 Writing 12.5 19.93 -7.43 2.54 12.5 22.47 32.43 Reading 12.33 -22.43 -16.27 -10.1 -3.94 2.23 Writing 9.68 High School Achievement Profile Technical Manual Arizona Department of Education•69 Appendix 5: Graduation and Dropout Rates Targets and Values Baseline* Dropout Rate 6-9% > 9% Target** 1% Decrease 2% Decrease Baseline* Graduation Rate Target** 74-90% 1% Increase < 74% 2% Increase School met target in Graduation Dropout 90% or Greater 6% or Less 3-Year 3-Year Average Average Yes Yes Yes No No Yes No No Subject/grade Value 7 5 3 3 0 * The baseline is the 2000 academic year. The annual dropout rate targets are the differences between the baseline year and the three year average for the 2000-2002 academic years. The graduation rate targets are the differences between the baseline year and the two year average for the 2000-2001 academic years. ** Achievement Profile Technical Manual Arizona Department of Education•70 Appendix 6: Subject Grade/Level Table For each subject/grade combination: If the AIMS three-year average is greater than or equal to 90%, the school shall receive 7 points for the subject/grade combination. If the baseline group is 1 and the total growth point value is less than the half standard deviation cut point (.5 SD), the school shall receive 0 points for the subject/grade combination. If the baseline group is 1 and the total growth point value is greater than or equal to the half standard deviation cut point (.5 SD), the school shall receive 5 points for the subject/grade combination. If the baseline group is 2 and the total growth point value is less than the mean cut point, the school shall receive 0 points for the subject/grade combination. If the baseline group is 2 and the total growth point value is greater than or equal to the mean cut point and less than the half standard deviation cut point (.5 SD), the school shall receive 3 points for the subject/grade combination. If the baseline group is 2 and the total growth point value is greater than or equal to the half standard deviation cut point (.5 SD), the school shall receive 5 points for the subject/grade combination. If the baseline group is 3 and the total growth point value is less than the negative half standard deviation cut point (-.5), the school shall receive 0 points for the subject/grade combination. If the baseline group is 3 and the total growth point value is greater than or equal to the negative half standard deviation cut point (-.5) and less than the half standard deviation cut point (.5 SD), the school shall receive 3 points for the subject/grade combination. If the baseline group is 3 and the total growth point value is greater than or equal to the half standard deviation cut point (.5 SD), the school shall receive 5 points for the subject/grade combination. If the baseline group is 4 or 5 and the total growth point value is less than the negative 1 standard deviation cut point (-1 SD), the school shall receive 0 points for the subject/grade combination. If the baseline group is 4 or 5 and the total growth point value is greater than or equal to the negative 1 standard deviation cut point (1 SD) and less than the mean cut point, the school shall receive 3 points for the subject/grade combination. If the baseline group is 4 or 5 and the total growth point value is greater than or equal to the mean cut point, the school shall receive 5 points for the subject/grade combination. Achievement Profile Technical Manual Arizona Department of Education•71 Appendix 7: Alternative School Definition A public school desiring identification as an alternative school must apply to the Arizona State Board of Education for such status. These schools must be separate entities according to Arizona school finance provisions (funded as a school, reported as a school, etc.). Alternative school status will not be granted to a program within a school.. An alternative school is a school that the Arizona State Board of Education has determined meets all of the following criteria: A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion #3) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. (Note: The school’s mission statement or charter must be communicated to the public.) The educational program and related student services of the school must match the mission or charter of the school. The school must intend to serve students exclusively in one or more of the following categories: Students with behavioral issues (documented history of disruptive behavior) Students identified as dropouts Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades Pregnant and/or parenting students Adjudicated youth Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Please Note: No public school district may have more than ten percent (10%) of their total student population attending an alternative school or any combination of alternative schools served by the district at one time. Smaller districts, if they wish, may participate in the development of a “consortium” alternative school. Achievement Profile Technical Manual Arizona Department of Education•72 Appendix 7 (Cont’d): Achievement Profile for Alternative Schools Ninety-five percent (95%) of students taking Arizona’s Instrument to Measure Standards (AIMS): Criteria: The ADE will develop a consistent formula to determine the percentage of students taking AIMS for all public schools and will apply this formula to alternative schools. Decrease Dropout rate: Criteria: Alternative schools will have the same Annual Dropout Rate targets as conventional public schools (see Table 2). Increase the percentage of graduates who demonstrate proficiency on the Standards via AIMS: Criteria: The 2002 academic year is the baseline. Every alternative school is expected to have 100% of graduates demonstrate proficiency of the Standards via AIMS by 2006. The expected annual progress for each alternative school is calculated as follows: Subtract the percentage of graduates who also demonstrate proficiency of the Standards on the 2000 AIMS from 100%. Divide the remainder by four (4). Achievement Profile Technical Manual Arizona Department of Education•73 Appendix 8: Small Schools Adjustment A small school is defined in ARS § 15-241 as a school with a student count of less than 100 as determined by unweighted Average Daily Membership (ADM). For every subject/grade combination with 16 or more students tested, the ADE will adjust the school’s data to remove low-performing outlier students and complete the conventional process to produce an achievement profile. If after the adjustment the subject/grade value is changed, the school shall receive the higher of subject/grade value. Achievement Profile Technical Manual Arizona Department of Education•74 Appendix 9: Subject/Grade Value Classification ScalesAll Grades 2 Subject/Grade Values: SCALE U M I E 0 5 10 14 4 9 13 14 3 Subject/Grade Values: SCALE U M I E 0 6 12 21 5 11 20 21 4 Subject/Grade Values*: SCALE U M I E 0 6 15 28 5 14 27 28 5 Subject/Grade Values**: SCALE U M I E 0 9 19 35 8 18 34 35 * A school in the 4 subject/grade values, with a point value of 7, consisting of three 0s and one 7, shall be underperforming. ** A school in the 5 subject/grade values, with a point value of 10, consisting of three 0s, one 3 and one 7, shall be underperforming. A school in the 5 subject/grade values, with a point value of 10, consisting of three 0s and two 5s shall be underperforming. A school in the 5 subject/grade values, with a point value of 12, consisting of three 0s, one 5 and one 7, shall be underperforming. Achievement Profile Technical Manual Arizona Department of Education•75 Appendix 9 (Cont’d): 6 Subject/Grade Values*: SCALE U M I E 0 9 24 42 8 23 41 42 7 Subject/Grade Values**: SCALE U M I E 0 12 26 49 11 25 48 49 8 Subject/Grade Values***: SCALE U M I E 0 12 29 56 11 28 55 56 * A school in the 6 subject/grade values, with a point value of 10, consisting of four 0s, one 3 and one 7, shall be underperforming. A school in the 6 subject/grade values, with a point value of 10, consisting of four 0s and two 5s shall be underperforming. A school in the 6 subject/grade values, with a point value of 12, consisting of four 0s, one 5 and one 7, shall be underperforming. ** A school in the 7 subject/grade values, with a point value of 12, consisting of five 0s, one 5 and one 7, shall be underperforming. *** A school in the 8 subject/grade values, with a point value of 12, consisting of six 0s, one 5 and one 7, shall be underperforming. Achievement Profile Technical Manual Arizona Department of Education•76 Appendix 9 (Cont’d): 9 Subject/Grade Values*: SCALE U M I E 0 15 36 63 14 35 62 63 * A school in the 9 subject/grade values, with a point value of 15, consisting of six 0s, one 3, one 5 and one 7, shall be underperforming. A school in the 9 subject/grade values, with a point value of 15, consisting of six 0s and three 5s, shall be underperforming. A school in the 9 subject/grade values, with a point value of 17, consisting of six 0s, one 3 and two 7s, shall be underperforming. A school in the 9 subject/grade values, with a point value of 17, consisting of six 0s, two 5s and one 7, shall be underperforming. A school in the 9 subject/grade values, with a point value of 19, consisting of six 0s, one 5 and two 7s, shall be underperforming. A school in the 9 subject/grade values, with a point value of 35, consisting of two 0s, seven 5 and no 7s, shall be improving. Achievement Profile Technical Manual Arizona Department of Education•77 Appendix 10: State Board Information Packet On September 23, 2002, the Arizona State Board of Education (Board) must adopt the achievement profile formula for Arizona public schools, including charter schools, in order to complete the school classifications as required in ARS § 15-241 (Arizona LEARNS) on October 15, 2002. The purpose of this document is to inform the Board of all the necessary decisions required to adopt the achievement profile and provide specific data to inform those decisions. This document includes an overview of the general process to produce the achievement profiles, a summary of the actions before the Board, specific numeric values associated with those actions and the administrative policies necessary to implement the achievement profiles. The achievement profile was developed according to a researchbased methodology by Arizona Department of Education (ADE) staff and members of the education community. The ADE will produce a technical report with specific formulas and supporting documentation. GENERAL PROCESS TO PRODUCE THE ACHIEVEMENT PROFILES The achievement profile for a public school includes a school classification and all related school improvement data. The general process to determine the achievement profile for each school is as follows: Identify the Baseline Group for each subject/grade combination Calculate total Growth Points for each subject/grade combination Determine the subject/grade value for each subject/grade combination Add all subject/grade values Evaluate the sum of subject/grade values according to the appropriate school classification scale Identifying the Baseline Group There are six (6) Baseline Groups created by five (5) different separation points. The separation points for each subject/grade combination are listed in Attachment One. Schools in Baseline Group 1 are below the Maintaining Line and can be classified only as either Under-performing or Improving. Achievement Profile Technical Manual Arizona Department of Education•78 BOARD ACTION: The Board must adopt the Baseline Group separation points in Attachment One. Calculating Total Growth Points Total Growth Points for each school and subject/grade combination are calculated by adding the following figures: Elementary schools (Reading and Mathematics): The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS over the 2000-2002 academic years and the percentage of students in the FFB performance level on the 2000 AIMS. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS over the 2000-2002 academic years and the percentage of students in the M/E performance levels on the 2000 AIMS. The Added Evidence Growth Points according to the average percentage of students making One Year’s Growth (OYG) according to the Measure of Academic Progress (MAP) over the 2000-2002 academic years (see Attachment 2). BOARD ACTION: The Board must adopt the Added Evidence Growth Points for MAP in Attachment 2. Elementary schools (Writing) The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2000-2002 academic years and the percentage of students in the FFB performance level on the 2000 AIMS. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2000-2002 academic years and the percentage of students in the M/E performance levels on the 2000 AIMS. The Added Evidence Growth Points based on the average percentage of students with an extended writing trait score of 24 or higher on AIMS over the 2000-2002 academic years (see Attachment 3). BOARD ACTION: The Board must adopt the Added Evidence Growth Points for extended writing in Attachment 3. High school (Reading and Writing) The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS Achievement Profile Technical Manual Arizona Department of Education•79 averaged over the 2000-2002 academic years and the percentage of students in the FFB performance level on the 2000 AIMS. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2000-2002 academic years and the percentage of students in the M/E performance levels on the 2000 AIMS. High school (Math) The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2001-2002 academic years and the percentage of students in the FFB performance level on the 2001 AIMS. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2002 academic years and the percentage of students in the M/E performance levels on the 2001 AIMS. Determining the Subject/Grade Value for Each Subject/Grade Combination There are four possible subject/grade values zero (0), three (3), five (5) and seven (7). The determination of subject/grade values is based on the following table. Table 1: Baseline Group 7 6 5 4 3 2 1 0 0 0 0 0 3 3 0 0 0 -1.0 SD 3 3 3 0 0 -.5 SD 5 5 3 3 0 _ X 5 5 5 5 5 .5 SD 5 5 5 5 5 1.0 SD A school with a three-year average of 90% of students Meeting or Exceeding the standards or higher in any subject/grade combination is awarded a subject/grade value of seven (7) for that subject/grade combination. Achievement Profile Technical Manual Arizona Department of Education•80 Maintaining Line The growth cut points for each subject/grade combination are listed in Attachment 4. BOARD ACTION: The Board must adopt the Growth Cut Points for each subject/grade combination in Attachment 4. The achievement profile for high schools includes the Graduation and Annual Dropout Rates. The following table summarizes the Graduation and Dropout Rate targets. Table 2: Baseline* Baseline* Dropout Rate Target** Graduation Rate Target** 6-9% 1% Decrease 74-90% 1% Increase > 9% 2% Decrease < 74% 2% Increase * The baseline is the 2000 academic year ** The Annual Dropout Rate targets are the difference between the baseline year And the three-year average for the 2000-2002 academic years. The Graduation Rate targets are the difference between the baseline year and the two-year average for the 2000-2001 academic years. BOARD ACTION: The Board must adopt the Graduation and Dropout Rate targets. Add All Subject/Grade Values The subject/grade values for the Annual Dropout and Graduation Rate indicators will be awarded based on the following table. Table 3: School met target in Subject/grade Graduation Dropout Value 90% or Greater 6% or Less 3-Year Average 3-Year Average 7 Achievement Profile Technical Manual Yes Yes 5 Yes No 3 No Yes 3 No No 0 Arizona Department of Education•81 BOARD ACTION: The Board must adopt a method to assign a grade/subject value to the Annual Dropout and Graduation Rate indicators according to whether a school has met the targets. Evaluating the Sum of Subject/Grade Values to Determine the School Classification The subject/grade values are added to derive a sum for each school. The sum of all subject/grade values is evaluated according to the appropriate school classification scale to determine the school classification. The ADE has created two options to assist the Board in adopting the school classification scales. These options are based on pivotal cases where the Board must provide guidance. The entire school classification scale can be developed based on the Board’s decision in these pivotal cases. The key policy question before the Board is whether the school classification scales should reflect a more compensatory or a more conjunctive methodology. In a compensatory approach, higher subject/grade values can have enough influence to improve the school classification despite the presence of low subject/grade values. In a conjunctive approach, higher subject/values are less likely to compensate for the presence of low subject/grade values. There are two pivotal cases where the Board must provide guidance. The first pivotal case involves breaking ties in cases where there are an equal number of adjacent subject/values. The second pivotal case involves the extreme scenario where a few number of the highest subject/grade value (7) can compensate for a larger number of the lowest subject/grade values (0). The following section includes examples and impact data for both the compensatory and conjunctive options. The Board may decide to break ties using the compensatory method and not allow extreme high values to offset the lowest values using the conjunctive method or vice versa. COMPENSATORY MODEL Under this option, schools with an even number of subject/grade values and an equal number of adjacent values are given the higher classification, with the exception of the Excelling classification. The following example is the case where a school has six (6) subject/grade values, but the example is applicable to any even number of subject/grade values. Achievement Profile Technical Manual Arizona Department of Education•82 Table 4: Grade/subject Values Class. 0 0 0 3 3 3 M 3 3 3 5 5 5 I 5 5 5 7 7 7 I In addition, the presence of at least one of the highest subject/grade values (7) can compensate for a larger number of the lowest subject/grade values (0). The following example is in the case where a school has three (3) subject/grade value but is generalizable to other odd number of subject/grade values. Table 5: Grade/subject Values Label 0 0 7 M 0 0 5 U School classification scales. The school classification scales under the compensatory option for the six (6) subject/value and three (3) subject value cases are as follows: Table 6: Six subject/grade values Classification Scale Underperforming 0 8 Maintaining 9 23 Improving 24 41 Excelling 42 42 Table 7: Three subject/grade values Classification Achievement Profile Technical Manual Scale Underperforming 0 5 Maintaining 6 11 Improving 12 20 Excelling 21 21 Arizona Department of Education•83 The school classification scales for the other subject/grade value combinations conform to the decisions consistent with the compensatory option and will be presented at the Board meeting. Impact data. The impact data is an estimate. The impact data must be interpreted with the following cautions: Alternative schools are included Small schools have not been adjusted to remove low-performing outlier students Not all extremely small schools have been excluded Only schools with all necessary data are included (missing data has not been adjusted to a subject/grade value of zero and irregular cases have not been corrected) Table 8: Compensatory Option: State level results Number of Subject/grade values 3 Total 4 6 9 Total Percent Underperforming 65 43 69 23 200 18% Maintaining 98 132 232 85 547 48% Improving 82 48 228 27 385 34% Excelling 0 0 0 0 0 0% Totals 245 223 529 135 1132 100% Note: All cells represent the number of schools with the exception of the last column. CONJUNCTIVE MODEL Under this option, schools with an even number of subject/grade values and an equal number of adjacent values are given the lower classification. The following example is the case where a school has six (6) subject/grade values, but the example is applicable to any even number of subject/grade values. Table 9: Grade/subject Values Achievement Profile Technical Manual Class. 0 0 0 3 3 3 U 3 3 3 5 5 5 M 5 5 5 7 7 7 I Arizona Department of Education•84 In addition, the presence of at least one of the highest subject/grade values (7) can not compensate for a larger number of the lowest subject/grade values (0). The following example is in the case where a school has three (3) subject/grade values, but is generalizable to other odd number of subject/grade values. Table 10: Grade/subject Values Label 0 0 7 U 0 0 5 U School classification scales. The school classification scales under the conjunctive option for the six (6) subject/value and three (3) subject value cases are as follows: Table 11: Six subject/grade values Classification Scale Underperforming 0 14 Maintaining 15 24 Improving 25 41 Excelling 42 42 Table 12: Three subject/grade values Classification Scale Underperforming 0 8 Maintaining 9 12 Improving 13 20 Excelling 21 21 The school classification scales for the other subject/grade value combinations conform to the decisions consistent with the conjunctive option and will be presented at the Board meeting. Impact data. The impact data is an estimate. The same cautions listed in the compensatory options also apply to the impact data in the conjunctive option. Achievement Profile Technical Manual Arizona Department of Education•85 Table 13: Conjunctive Option: State level results Number of Subject/grade values 3 Total 4 6 9 Total Percent Underperforming 91 68 134 51 344 30% Maintaining 73 120 201 65 459 41% Improving 81 35 194 19 329 29% Excelling 0 0 0 0 0 0% Totals 245 223 529 135 1132 100% Note: All cells represent the number of schools with the exception of the last column. BOARD ACTION: The Board must adopt a school classification scale for every possible subject/grade combination. The Arizona Department of Education has provided two different sets of school classification scales and the Board may adopt one. PROCESS TO IDENTIFY UNIQUE SCHOOLS Small Schools Adjustment A small school is defined in ARS § 15-241 as a school with a student count of less than 100 as determined by unweighted Average Daily Membership (ADM). For every subject/grade combination with 16 or more students tested, the ADE will adjust the school’s data to remove low-performing outlier students and complete the conventional process to produce an achievement profile. If after the adjustment the subject/grade value is changed, the school shall receive the higher of subject/grade value. Alternative Schools A public school desiring identification as an alternative school must apply to the Arizona State Board of Education for such status. These schools must be separate entities according to Arizona school finance provisions (funded as a school, reported as a school, etc.). Alternative school status will not be granted to a program within a school. Alternative School Definition An alternative school is a school that the Arizona State Board of Education has determined meets all of the following criteria: A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion #3) that will benefit from an alternative school setting. A charter Achievement Profile Technical Manual Arizona Department of Education•86 school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. (Note: The school’s mission statement or charter must be communicated to the public.) The educational program and related student services of the school must match the mission or charter of the school. The school must intend to serve students exclusively in one or more of the following categories: Students with behavioral issues (documented history of disruptive behavior) Students identified as dropouts Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades Pregnant and/or parenting students Adjudicated youth Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Please Note: No public school district may have more than ten percent (10%) of their total student population attending an alternative school or any combination of alternative schools served by the district at one time. Smaller districts, if they wish, may participate in the development of a “consortium” alternative school. Achievement Profile for Alternative Schools Ninety-five percent (95%) of students taking Arizona’s Instrument to Measure Standards (AIMS). Criteria: The ADE will develop a consistent formula to determine the percentage of students taking AIMS for all public schools and will apply this formula to alternative schools. Decrease Dropout rate. Criteria: Alternative schools will have the same Annual Dropout Rate targets as conventional public schools (see Table 2). Increase the percentage of graduates who demonstrate proficiency on the Standards via AIMS. Criteria: The 2002 academic year is the baseline. Every alternative school is expected to have 100% of graduates demonstrate proficiency of the Standards via AIMS by 2006. The expected annual progress for each alternative school is calculated as follows: Achievement Profile Technical Manual Arizona Department of Education•87 1. Subtract the percentage of graduates who also demonstrate proficiency of the Standards on the 2000 AIMS from 100%. 2. Divide the remainder by four (4). The ADE will report the progress of all alternative schools to the Board in Fall 2003. The first school classification for alternative schools will be provided in Fall 2004. BOARD ACTION: The Board must approve the definition, criteria and timeline for the alternative schools achievement profile. ADMINISTRATIVE POLICIES Schools Not Receiving a Label on October 15, 2002 New schools – defined as schools that opened for the first time after Summer 2000. Once a school has been operational for three (3) test administrations, the school will receive an achievement profile. K-2 schools – defined as schools that serve any combination of grades from kindergarten to second grade and do not serve students in grades three (3) or higher. Accommodation schools – see statute for definition Extremely small schools – defined as schools with less than 16 students in over 1/3 of all possible subject/grade combinations. Schools with at least 16 students in 2/3 or more of all possible subject/grade combinations will receive an achievement profile based on the data available. Alternative schools – includes all schools that have indicated their intention to apply to the Board for alternative school status. If a school is not granted alternative school status, the school will be evaluated according to the conventional achievement profile process adopted by the Board for other schools that serve students in similar grade levels (See the Alternative School section for the achievement profile criteria and formula for schools granted alternative school status by the Board). Missing Data A school that has not provided the necessary data for any subject/grade combination shall receive a subject/grade value of zero (0) for that subject/grade combination. BOARD ACTION: The Board must approve the preceding actions/policy decisions. Achievement Profile Technical Manual Arizona Department of Education•88 REVIEW The Board shall review the achievement profile formula on an annual basis. Achievement Profile Technical Manual Arizona Department of Education•89