Arizona Department of Child Safety Douglas A. Ducey Gregory McKay Governor Director May 10, 2016 The Honorable Andy Biggs President of the Senate Arizona State Senate 1700 West Washington Phoenix Arizona 85007 Re: Arizona Families F.I.R.S.T. 2015 Annual Evaluation Report Dear President Biggs: Pursuant to A.R.S. § 8-884, the Arizona Department of Child Safety (DCS) is pleased to provide the Arizona Families F.I.R.S.T. (AFF) program evaluation report for the state fiscal year 2015 (SFY 2015). This annual evaluation report prepared by the independent evaluator, Arizona State University (ASU) / Center for Applied Behavioral Health Policy (CABHP), provides analysis of program implementation, service utilization and program outcome data. After receiving the first draft of this report from ASU, the Department's previous administration requested that ASU complete further and additional analysis of the data. Highlights contained within the enclosed report include:  During the SFY 2015, 7,388 unique individuals were referred to AFF program, representing a 34% percent increase from SFY 2014.  The total number of individuals that received services during SFY 2015 was 7,721.  98.8% of new referrals received some form of outreach and the majority (92.4%) received within one day or less.  In SFY 2015, 77.3% percent of AFF referrals received in SFY 2014 had no subsequent maltreatment allegation 12 months or more following AFF closure.. If you have any questions, please contact my office at (602) 255-2500. P.O. Box 6030  Site Code C010-23  Phoenix, AZ 85005-6030 Telephone (602) 255-2500 The Honorable Andy Biggs Arizona Families F.I.R.S.T. 2015 Annual Evaluation Report Page 2 Sincerely, Gregory McKay Director Enclosures cc: Speaker David M. Gowan, Speaker, Arizona State House of Representatives Representative John Allen, Chairperson, Joint Legislative Audit Committee Representative Regina Cobb, Member, Joint Legislative Audit Committee Representative Debbie McCune Davis, Member, Joint Legislative Audit Committee Representative Rebecca Rios, Member, Joint Legislative Audit Committee Representative Kelly Townsend, Member, Joint Legislative Audit Committee Senator Judy Burges, Member, Joint Legislative Audit Committee Senator Nancy Barto, Member, Joint Legislative Audit Committee Senator Lupe Contreras, Member, Joint Legislative Audit Committee Senator David Farnsworth, Member, Joint Legislative Audit Committee Senator Lynne Pancrazi, Member, Joint Legislative Audit Committee Joan Clark, Director, Arizona State Library, Archives and Public Records Arizona Families F.I.R.S.T. Program Annual Evaluation Report State Fiscal Year 2015 Prepared for: Department of Child Safety Phoenix, Arizona Prepared By: Center for Applied Behavioral Health Policy College of Public Service and Community Solutions Arizona State University Date February 19, 2016 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Acknowledgements This report was prepared by the Arizona State University Center for Applied Behavioral Health Policy (ASU CABHP), under contract number DES060718-001 with the Department of Child Safety (DCS), in partnership with the Arizona Department of Health Services, Division of Behavioral Health Services (DHS) through the Joint Substance Abuse Treatment Fund. The authors wish to thank the following staff for their ongoing cooperation and assistance: Jenna Shroyer, Nicolas Espadas, William Aldrich, James Woods, Antonella Manetti, and Leanne Hawkins of DCS; Patrick Birmingham, Martha Alabado, Delmar Zahnleiter, Rosemary Celaya, Donna Ruiz, and Li Kuohsiung of the Arizona Department of Economic Security (DES); and Anne C. Dye, Jacqueline Picone, Marilyn Fields, Madonna R. Fritz, and Paul J. Galdys of DHS. Finally, the authors wish to express their appreciation to the staff and directors of the contracted Arizona Families F.I.R.S.T. (AFF) provider agencies throughout the state. We appreciate their insights and recommended strategies for improving the utility and quality of the information contained in this report. Most notably, we recognize and honor their dedication to the families and children served through the AFF program. Points of view represented in this report are those of the authors and do not necessarily represent the official position of either the DCS or the DHS. Suggested citation: Shafer, M. S., Sayrs, L. W., Rivera, R., Harootunian, G., & Mendoza, N. (2015). Arizona Families F.I.R.S.T. Program: Annual Evaluation Report for the Period July 1, 2014 – June 30, 2015. Phoenix, AZ. Arizona State University. II Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Executive Summary This report is issued by the Arizona State University Center for Applied Behavioral Health Policy and reports on the patterns of referrals, services, and outcomes associated with Arizona Families F.I.R.S.T. (Families in Recovery Succeeding Together; AFF) for State Fiscal Year 2015 (July 1, 2014 – June 30, 2015). AFF was established in 2000 to address adverse conditions related to alcohol and drug abuse among child-welfare-involved families in which allegations of child maltreatment were associated with parental substance abuse. The AFF program provides a variety of treatment and supportive services designed to reduce or eliminate abuse of, and dependence on, alcohol and other drugs in family systems. Interventions are provided through the Arizona Department of Child Safety (DCS), contracted community providers in outpatient and residential settings, and/or through the Regional Behavioral Health Authority (RBHA) provider network under contract with the Arizona Department of Health Services, Division of Behavioral Health Services (DHS). During this reporting period the contracted AFF providers were Arizona Partnership for Children, Southeastern Arizona Behavioral Health Services, and Terros. Key elements of the AFF program include: an emphasis on face-to-face outreach and engagement at the time of program referral; assessments; supportive services (e.g., case management, transportation, employment, and housing services); individual and group treatment; counseling; and recovery maintenance services. The service-delivery model incorporates essential elements based on family needs, such as culturally responsive services, gender-specific treatment, family-involved treatment services, and motivational enhancement strategies to assist the entire family in its recovery. Referral Patterns and Characteristics of Individuals Referred for AFF Services • 8,562 new referrals for AFF services (representing 7,388 unique individuals) were reported in SFY 2015, representing a 34% increase in referrals from the 6,419 that were received in SFY 2014. • 7,486 referrals were closed, representing a 63% increase over the 4,606 referrals that had been closed in SFY 2014. • 52.6% of the individuals referred to the AFF program were 30 years of age or younger. Two thirds (62.5%) were female, and the majority were single, never married, white, and non-Hispanic. • Approximately 70% of all new individuals referred to the AFF program in SFY 2015 had a substantiated or unsubstantiated allegation of child maltreatment. • 90% of the 3,753 unique individuals for whom substance abuse assessment results were provided reported use in the 30 days prior to their AFF referral. • 21.5% identified heroin/opiates or other opiates/sedatives as their primary substance of abuse. III Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Timeliness, Availability, and Accessibility of Services • 98.8% of new referrals received some form of outreach and the majority (92.4%) received it within one day or less. • 50.9% of referrals provided a release of information within three weeks (21 days median) after referral. • 7,721 individuals received AFF services; 63.2% had been referred to the program in this fiscal year, while an additional 29% had been referred in SFY 2014 but continued to receive services for some period of time in SFY 2015. • AFF services were jointly funded by both DCS and DHS for 45% of all clients, with 32.8% of AFF clients receiving all of their services funded solely by DHS, and 22.2% receiving all of their services funded solely by DCS. • “Other” (96.4%) represented the most commonly reported AFF service and the most commonly reported substance abuse treatment service (98.9%) among those individuals with DCS-funded services. • Support (95.4%) and treatment (87.1%) services were the more commonly reported services funded by DHS. • 61.7% of all clients with AFF services had at least one level of care assignment during their AFF treatment. • Outpatient (77.4%) and intensive outpatient (31.5%) were the more commonly reported levels of care among those individuals for whom level of care was reported. • 7,486 closures were reported from the 12,188 new and continuing referrals this year with the median duration of program participation of 98 days (mean=143.4, SD=126.4). • 864 referrals (11.5% of all closures) were closed as “Completed AFF at the Conclusion of SA Treatment,” with a median duration of program completion of 335 days (mean=312.8, SD=147.8). • “Unable to locate/refused services” represented the single largest category of reported closures, representing 4,954 (67%) of all reported closures. • “Discontinued” represented the most commonly reported closure reason among referrals that had been engaged in AFF services, representing 21.7% of closures with services. • “Completion” was reported for 11.54% of all reported closures with services. Recovery from Alcohol and Drug Problems • 65.8% (n=5,081) of individuals who received services were referred at least once for drug testing, resulting in 64,238 drug test referrals. • On average, individuals were drug tested 2.8 times per month (median 2.02/month). • 84.9% of all usable drug test results indicated no drug use. Employment Outcomes • Individuals who completed AFF services demonstrated significantly higher rates of employment at intake when compared to individuals who discontinued AFF services (40.2% vs. 28.6%). IV Arizona Families F.I.R.S.T. • Annual Evaluation Report SFY 2015 Significantly more individuals who completed AFF services had become employed at closure compared to individuals to discontinued AFF services (52.4% vs. 32.7%). Permanency • 3,066 (48%) children had achieved permanency out of the 6,383 children that had been placed in out-of-home care that were associated with parents who were closed from AFF services in SFY 2014. • Among those children that achieved permanency, 55.6% did so through reunification, 36.4% through adoption, and 7.8% through guardianship. • Children whose parents had completed AFF services experienced a significantly higher rate of permanency than children whose parents had discontinued AFF services (65.5% vs. 40.0%, or 699 vs. 648). • Children whose parents had completed AFF services experienced a significantly higher rate of reunification than children whose parents had discounted AFF services (84.8% vs. 52.6%, or 593 vs. 341). Maltreatment Re-Occurrence • 77.3% of all AFF referrals that were processed in SFY 2014 and subsequently closed had no subsequent maltreatment allegation 12 months or more following AFF closure. • New maltreatment allegation filings did not vary by whether the client completed AFF services or discontinued AFF services. V Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Table of Contents Acknowledgements .................................................................................................................................... II Executive Summary ................................................................................................................................... III Table of Contents .......................................................................................................................................VI List of Exhibits ..........................................................................................................................................VII 1.0 Introduction .................................................................................................................................... 1 2.0 Evaluation Framework and Data Sources ................................................................................... 5 3.0 Individuals and Services Received .............................................................................................. 9 3.1 Patterns and Characteristics of Referrals for Services ....................................................... 9 3.2 Outreach and Engagement in Services ............................................................................ 12 3.3 Patterns of Substance Abuse among Individuals Referred for Services .......................... 13 3.4 Patterns of Service Delivery .............................................................................................. 15 3.5 Patterns of Program Closure and Length of Service ........................................................ 19 4.0 Program Outcomes ...................................................................................................................... 21 4.1 Patterns and Results of Drug Testing ............................................................................... 21 4.2 Employment ...................................................................................................................... 21 4.3 Longitudinal Analysis of Permanency ............................................................................... 23 4.4 Longitudinal Analysis of Maltreatment Recidivism ............................................................ 24 5.0 Key Findings and Program Implications ................................................................................... 27 5.1 Child Permanency and Reunification ................................................................................ 27 5.2 Child Safety ....................................................................................................................... 28 5.3 Parental Substance Abuse ............................................................................................... 29 5.4 Parental Employment ........................................................................................................ 30 5.5 Availability, Timeliness, and Accessibility of Substance Abuse Treatment Services ....... 30 5.6 Conclusions, Implications, and Next Steps ....................................................................... 31 VI Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 List of Exhibits Exhibit 1. Overview of the AFF Program Model, SFY 2015 .......................................................................... 2 Exhibit 2. Map of AFF Provider Regions 2015 .............................................................................................. 3 Exhibit 3. List of DCS Regions, Counties, DCS Providers, and RBHAs, SFY 2015..................................... 4 Exhibit 4. AFF Referrals, SFY 2015 .............................................................................................................. 9 Exhibit 5. AFF Total Referrals and Unique Individuals by Quarter, SFY 2011 – SFY 2015 ....................... 10 Exhibit 6. AFF Demographics for Individuals, SFY 2015 ............................................................................ 11 Exhibit 7. Maltreatment Allegations for Unique Individuals with New Referrals, SFY 2015 ....................... 12 Exhibit 8. Outreach, SFY 2015 ................................................................................................................... 12 Exhibit 9. AFF Acceptance of Services, SFY 2015 ..................................................................................... 13 Exhibit 10. Unique Individuals with Substance Abuse Assessment ........................................................... 14 Exhibit 11. AFF Patterns of Self-Reported Substance Abuse, SFY 2015 .................................................. 15 Exhibit 12. Total Unique Individuals Referred and Served, SFY 2015 ....................................................... 16 Exhibit 13. Total Unique Individuals Served and Funding Source, SFY 2015 ............................................ 16 Exhibit 14. Individuals Receiving DCS-Funded Services, SFY 2015 ......................................................... 17 Exhibit 15. Individuals Receiving DHS-Funded Services, SFY 2015 ......................................................... 18 Exhibit 16. Level of Care, SFY 2015 ........................................................................................................... 19 Exhibit 17. AFF Closures and Length of AFF Participation, SFY 2015 ...................................................... 20 Exhibit 18. Patterns and Results of Drug Testing, SFY 2015 ..................................................................... 21 Exhibit 19. Employment Outcomes, SFY 2015 ........................................................................................... 22 Exhibit 20. Permanency and Reunification, SFY 2014 Cohort ................................................................... 23 Exhibit 21. Maltreatment Recurrence, SFY 2014 Cohort Longitudinal Analysis ......................................... 25 VII Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 1.0 Introduction Arizona Families F.I.R.S.T. (AFF) was established as a community substance use disorder prevention and treatment program by Senate Bill 1280, which passed in the 2000 legislative session. The Joint Substance Abuse Treatment Fund established by this legislation requires an annual evaluation of the AFF program. This evaluation examines the implementation and outcomes of community substance use disorder treatment services delivered by providers contracted with Arizona Department of Child Safety (DCS). The AFF program provides contracted, family-centered, strengths-based, substance abuse treatment and recovery support services to parents or caregivers whose substance abuse is a significant barrier to maintaining or reunifying the family, or to maintaining employment. Individuals are referred by Child Safety Specialists at DCS and by the Jobs Program (i.e., mandatory employment and training program for work-eligible individuals in households receiving cash assistance). The program seeks to eliminate abuse of, and dependence on, alcohol and other drugs, and to address other adverse conditions related to substance abuse. Interventions are provided through the DCS, contracted community providers with services provided in outpatient and residential settings, and/or the RBHA network of providers. In addition to traditional service, the AFF program emphasizes face-to-face outreach and rapid engagement at the beginning of services, supportive services to remove barriers (e.g., transportation and housing), and recovery maintenance to support ongoing sobriety and recovery. Service delivery incorporates essential elements based on family needs in conjunction with culturally responsive services, gender-specific treatment, motivational enhancement strategies, and collaboration with child service providers to assist the entire family in its recovery. The following three exhibits provide an overview of the AFF program model (Exhibit 1), a map of AFF provider regions (Exhibit 2), and a listing of DCS regions, counties, DCS providers, and RBHAs (Exhibit 3). Center for Applied Behavioral Health Policy Arizona State University | 1 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 1. Overview of the AFF Program Model, SFY 2015 Center for Applied Behavioral Health Policy Arizona State University | 2 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 2. Map of AFF Provider Regions 2015 Center for Applied Behavioral Health Policy Arizona State University | 3 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 3. List of DCS Regions, Counties, DCS Providers, and RBHAs, SFY 2015 DCS Region Central County Maricopa East Pinal Pima Southwest Southeast Northern Pima RBHA 2014 RBHA 2015 DCS Provider 2015 Mercy Maricopa Integrated Care (MMIC) Cenpatico Community Partnership of Southern Arizona (CPSA) Mercy Maricopa Integrated Care (MMIC) Cenpatico Terros Central Cenpatico Terros Pima Mercy Maricopa Integrated Care (MMIC) Terros Southwest Maricopa West Magellan Yuma La Paz Gila Cochise Graham Greenlee Santa Cruz Cenpatico Cenpatico Coconino Northern Arizona Regional Behavioral Health Authority (NARBHA) Health Choice Integrated Care (HCIC) Center for Applied Behavioral Health Policy Southern Arizona Behavioral Health Services (SEABHS) Arizona Partnership for Children (AzPaC) Arizona State University | 4 Arizona Families F.I.R.S.T. 2.0 Annual Evaluation Report SFY 2015 Evaluation Framework and Data Sources This evaluation report responds to the legislatively mandated performance indicators of the AFF program. The data provided herein are drawn from administrative data submitted to the evaluation team directly or obtained from administrative information files maintained by DCS and DHS. These data, like those reported in previous reports, include: • • • • Client characteristics and service utilization data obtained directly from the DCScontracted providers; Child maltreatment allegation and child out-of-home placement information obtained through DCS CHILDS (Children’s Information Library and Data Source); Enrollment and service utilization information for services provided through the RBHA network of providers obtained through the DHS CIS (Client Information System); and Jobs participation and Temporary Assistance for Needy Families (TANF) benefits information obtained through the DES/JAS/AZTEC (Jobs Automated System/Arizona Technical Eligibility Computer System). Beginning in SFY 2015, AFF providers began submitting data to ASU through a new data portal, following data specifications and criteria negotiated with, and approved by, the contracted service providers and DCS. Two primary factors necessitated the change in data reporting processes between the providers and ASU. First, the new service contract specifications that DCS promulgated with the SFY 2013 AFF contract awards were inadequately evaluated by the legacy reporting structures. Second, data transfer systems with more robust security were needed due to the sensitive HIPAA-level information that the providers were submitting to ASU. The most significant aspects of these altered procedures were the elimination of manual data entry through a web portal and the elimination of the providers’ and DCS’s ability to conduct web-based data look-ups on individual clients through ASU’s data resources. The new data upload specifications were finalized in the spring of 2014 and required all providers to upload data to ASU in nine data tables using data file formats that would ensure cross-agency consistency and lead to better data integrity. These data uploads are supposed to occur on a monthly basis. ASU received the first data upload for SFY 2015 from one provider in September 2014 and, by November 2014, ASU was receiving data uploads from all three providers. Upon receipt of a data upload, ASU evaluates the file structure of each table to ensure that it meets the specified standards. Both DCS and the contracted providers have been provided monthly reports on data upload activity and on the conformance of their files to data specifications (referred to as Data Upload Clearance Reports). These reports provide specific and actionable information that allow providers to correct errors in their data table formats. ASU issued its first data upload clearance reports in February 2015 and continued to issue these reports on a monthly basis throughout the balance of the SFY. In March 2015, one provider’s Center for Applied Behavioral Health Policy Arizona State University | 5 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 data uploads met file structure specifications; in April 2015, two providers met specifications; and as of June 2015, all three providers’ data tables met specification standards. ASU began focusing on data integrity as AFF providers met data file upload specifications. In May 2015, ASU began issuing detailed and actionable data integrity reports to the providers and to DCS. These monthly reports identify missing, out-of-range, and logically inconsistent data contained in specific fields in each of the nine submitted data tables. To ensure that the source of the data error or omission was being addressed, ASU requested that each provider deliver documentation identifying the corrections they had made in response to these reports. ASU also provided ongoing technical assistance to providers through scheduled phone calls to review the monthly integrity reports and to respond to any provider questions or concerns. On an annual basis, ASU provides a roster of clients, as reported by the providers to the DCS, DHS, and DES. These three state agencies use matching algorithms to identify individuals referred to AFF providers who also appear in their respective data systems. Where matches occur, the state agency extracts an agreed-upon set of data elements and transmits it to ASU in a secured format. ASU personnel then integrate data from these four data sources (providers, DCS, DHS, DES) to create the interoperable data set that serves as the basis for this report. Changes in the Analytic Approach Due to the previously referenced changes in specifications governing service and data uploading, the data analytic approaches undertaken by ASU in the production of this report have changed. Consistent with the analysis contained in the SFY 2014 report, as reissued, this report captures a 12-month period (July 1, 2014 – June 30, 2015). For SFY 2015, ASU extended the deadline for uploading data from June 30, 2015, to September 30, 2015, in order to provide additional time for providers to clean and upload data with better data integrity. For reporting of services provided only in the 2015 SFY (July 1, 2014 – June 30, 2015), ASU extended the data deadline further, to October 31, 2015. Some data contained in this report, particularly those data associated with service utilization, will likely underrepresent the true extent or volume of services provided. This is due to data entry lags in the agencies that generate these data (e.g., treatment providers and DCS field offices) and to the data reconciliation processes in their organizational systems (e.g., RBHA–DHS; DCS central office). The extent of this under-representation is not known at this time. For this SFY 2015 Annual Report, ASU introduces additional evaluation design elements that allow for an assessment of AFF program participation impact upon the legislatively mandated outcomes of child safety, permanency, and employment. These design elements include comparisons made between program completers and program drop-outs as well as a Center for Applied Behavioral Health Policy Arizona State University | 6 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 longitudinal analyses of post-AFF program patterns of permanency and maltreatment recidivism. New to this year’s analyses, four cohorts of AFF program participants have been created using closure codes submitted by treatment providers at the time of case closure. These four cohorts consist of those individuals who were closed completed (e.g., completed treatment), discontinued AFF services before treatment completion, could not be located by treatment referral after AFF referral or refused AFF services, and other (dead and incarcerated). Exhibit 17 provides a breakdown of the number of AFF referrals that were closed by grouping along with the duration of AFF participation. Comparisons are made in the employment outcomes between those cases that were closed as completed and those that were discontinued. These results are presented in Exhibit 19. The analysis of permanency and maltreatment recidivism outcomes reported in this year’s report was conducted on the group of individuals referred in SFY 2014 who were subsequently closed in SFY 2014 or SFY 2015. The reason for this change in methodology is the long-held recognition by officials at DCS and by the ASU evaluation team that previous reports of the AFF program underestimated the true rate of reunification and maltreatment recurrence. These underestimations were due to calculating these events (permanency and maltreatment reoccurrence) within the same state fiscal year in which an individual had been referred to the AFF program. As such, an individual who might be referred to the AFF program in May would only have two months in which these events could occur. Research indicates that, on average, permanency varies by type of exit from foster care (reunification, adoption, guardianship, other relative care) and ranges from under 30 days to a year for reunification, and typically over two years for adoption. 1,2,3 Maltreatment re-occurrence is less likely to be observed immediately following program participation than 6 to 12 months post program participation. 4 Consequently, analysis of the impact of the AFF program upon permanency and maltreatment recurrence is reported for those cases that were referred in SFY 2014 only. The SFY 2014 cohort was selected as the index cohort from which subsequent longitudinal analyses would be conducted based upon an assessment of the relative integrity of the data. Data integrity procedures implemented by the evaluation team during SFY 2015, along with the resubmission of data from the largest AFF contractor for the SFY 2014 Reissuance Report has provided enhancements in the quality and completeness of data. 1 Carnochan, S., Rizik-Baer, D., & Austin, M. J. (2013). Preventing the recurrence of maltreatment. Journal of Evidence-Based Social Work, 10(3), 161-178. 2 Akin, B. A. (2011). Predictors of foster care exits to permanency: A competing risks analysis of reunification, guardianship, and adoption. Children and Youth Services Review, 33(6), 999-1011. 3 McCombs-Thornton, K. L. (2011). Fostering a permanent home: A mixed methods evaluation of the Zero to Three Court Teams for maltreated infants and toddlers initiative (Doctoral dissertation). The University of North Carolina at Chapel Hill. 4 Fluke, J. D., Shusterman, G. R., Hollinshead, D. M., & Yuan, Y. Y. T. (2008). Longitudinal analysis of repeated child abuse reporting and victimization: Multistate analysis of associated factors. Child Maltreatment, 13(1), 76-88. Center for Applied Behavioral Health Policy Arizona State University | 7 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Comparisons between individuals’ AFF closure status (completed, discontinued, unable to locate/refused services, and other) are made for the two outcomes, permanency and maltreatment. Ostensibly, these comparisons, like those provided for employment outcomes, seek to test the hypothesis that individuals who are successfully treated in the AFF program experience better outcomes; in this case, higher rates of reunification, and lower rates of maltreatment recurrence than other individuals who did not successfully complete AFF services, were found. Slight variations in the number of individuals and children reported in the reissuance of the SFY 2014 report and these analyses are detected due to ongoing data submissions by providers and the well documented issues related the CHILDS data system. Data collected from SFY 2015 are not easily reconciled to prior years’ data, including the 2014 Annual Report, due to the changes noted above. As a result, extreme caution should be taken in comparing indicators from this years’ report and last years’ report. With the exception of the number of referrals to AFF and the number of referred individuals, no other comparisons are made within this report to SFY 2014 indicators. Subsequent annual analysis will allow for more valid year-over-year comparisons. Center for Applied Behavioral Health Policy Arizona State University | 8 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 3.0 3.1 Individuals and Services Received Patterns and Characteristics of Referrals for Services A total of 8,562 new referrals (representing 7,388 unique individuals) for AFF services were reported in SFY 2015, representing a 34% increase in referrals from the 6,419 5 that were received in SFY 2014. An additional 3,626 referrals (representing 3,624 unique individuals), originally received in SFY 2014, were carried over into SFY 2015, bringing the total number of referrals for AFF services that were processed in SFY 2015 to 12,188. These new and continuing referrals represent 10,330 unique individuals, since an individual can be referred more than once within a fiscal year. The number of unique individuals referred in SFY 2015 was 34% greater than the 7,692 individuals that were referred in SFY 2014. The rate of referrals remained relatively constant throughout the SFY, fluctuating between 2,039 and 2,382 processed each quarter. During this fiscal year, a total of 7,486 referrals were closed, representing a 63% increase over the 4,606 referrals that had been closed in SFY 2014. Exhibit 4. AFF Referrals, SFY 2015 SFY Totals 2015 New Referrals Continuing Referrals All Referrals Closed Referrals New and Continuing Unique Individuals 2014 New Referrals Continuing Referrals All Referrals N % 8562 64.6 SFY Carryforward 3626 4702 Quarter 4 Quarter 3 Quarter 2 Quarter 1 N % N % N % N % 2382 35.4 2091 34.4 2050 35.1 2039 36.0 4354 64.6 3980 65.6 3786 64.9 3626 64.0 12188 100.0 6736 7486 56.4 2034 30.2 1718 28.3 1856 31.8 1879 33.2 6458 62.5 5849 56.6 5596 54.2 5443 52.7 1831 33.1 1637 33.1 1446 33.8 1505 38.9 3704 66.9 3304 66.9 2832 66.2 2359 61.1 10330 6419 73.1 2359 26.9 8778 Closed Referrals 4606 New and Continuing Unique Individuals 7692 4172 6071 5535 52.5 5836 4941 5665 4278 3864 1363 24.6 1237 25.0 974 22.8 1032 26.7 5399 70.2 4754 61.8 4185 54.4 3726 48.4 5 The SFY 2014 AFF Annual Report Reissuance reported 7,272 referrals. AFF service contractors provide new data uploads to ASU that overwrite previous data uploads. ASU re-queries the AFF service contractor data to ensure that we are using the most accurate information. The discrepancy between SFY 2014 Report (7,272) and the current report of SFY 2014 referrals (6,419) is due to changes in AFF service contractor reporting. Center for Applied Behavioral Health Policy Arizona State University | 9 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 4. AFF Referrals, SFY 2015 – Statewide (Continued) Note. 1. “New Referrals” indicates referrals that were received and processed in the quarter or SFY, as indicated. The percentage of new referrals reflects the number of new referrals out of all referrals (new and carryforward) processed in the same time period 2. “Continuing Referrals” indicates referrals that had been received in a preceding quarter and had not been closed in that quarter. Since referrals may be carried forward over multiple quarters, the quarterly tallies of continuing referrals cannot be summed across all four quarters for an SFY total 3. “Closed Referrals” indicates referrals for which a discharge/closure report had been filed in the specified quarter. The percentage of closed referrals reflects the number of closed referrals out of all referrals (new and continuing) processed in the same period 4. “New and Continuing Individuals” refers to the number of persons for whom one or more referral-related activities had been processed in a specified time period 5. “SFY Carryforward Referrals” indicates non-closed referrals at the end of the fourth quarter that will be carried forward into the first quarter of the subsequent fiscal year 6. “SFY 2014 Carryforward Referrals” (n = 4,172) was reduced by 546 referrals for carryforward into SFY 2015, Quarter 1 (n=3,626). These referrals had been reported by the providers as carryforward into SFY 2014 and had been processed in 2013 or earlier. Due to significant issues in the data integrity associated with the longstanding open referrals and the lack of associated reported service activity, they were dropped from subsequent analyses Exhibit 5. AFF Total Referrals and Unique Individuals by Quarter, SFY 2011 – SFY 2015 Center for Applied Behavioral Health Policy Arizona State University | 10 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Just over one half (52.6%) of the individuals referred to the AFF program in SFY 2015 were 30 years of age or younger. Two thirds (62.5%) were female, and among those for whom the information was reported, the majority were single and had never been married (83%), white (26%), and non-Hispanic. Among those individuals for whom information was reported, nearly 20% (19.9%) had no higher than a high school diploma/GED level of education, and 31.6% were employed full- or part-time at the time of their referral for AFF services. Exhibit 6. AFF Demographics for Individuals, SFY 2015 Total (N=10,330) N % Age (years) < 18 18-24 25-30 31-35 36-45 46-55 > 55 Total Gender Female Male Total Marital Status Single, Never Married Divorced/Separated Widowed/Living Alone Married Domestic Partner/Cohabitation Total Race White Black American Indian Native Hawaiian Asian Other Unknown Total 23 2137 3271 2368 2001 465 64 10329 0.2 20.7 31.7 22.9 19.4 4.5 0.6 100.0 6452 3876 10328 62.5 37.5 100.0 3747 685 39 31 12 4514 83.0 15.2 0.9 0.7 0.3 100.0 2018 221 162 15 7 77 5399 7756 26.0 2.9 2.1 0.2 0.1 1.0 69.6 Total (N=10,330) N % Hispanic/Latino Ethnicity Yes No Unknown Total Language English Spanish Other Total Education Less than High School/GED High School Graduate or GED Vocational/Technical School Some College, No Degree College – AA/BA Degree Graduate or Post Graduate degree Unknown Total Employment Status Competitively Employed Full-Time Competitively Employed Part-Time Unemployed Other Unknown Total 700 9625 0 10325 6.8 93.2 0.0 100.0 9683 218 103 10004 96.8 2.2 1.0 100.0 2042 2056 180 1222 245 24 4547 10316 0.4 19.9 1.7 11.9 2.4 0.2 44.1 100.0 1272 661 2425 311 1452 6121 20.8 10.8 39.6 5.1 23.7 100.0 Note. 1. An individual may have two referrals, one which was closed and a separate referral that was not 2. Categories may not equal 10,330 (N) due to individuals with missing data or without an assessment 3. Age was calculated using the date of birth and the date of first referral 4. Race categories are not mutually exclusive. More than one category may have been selected per individual assessed, and the overall sum may exceed 100% Center for Applied Behavioral Health Policy Arizona State University | 11 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 At the time of referral to AFF, 81.4% of the 7,388 unique individuals referred in SFY had a substantiated or unsubstantiated allegation of child maltreatment. An additional 17.5% had a proposed maltreatment allegation, while 9.2% of the individuals referred to AFF in SFY 2014 had no recorded allegation of child maltreatment. Exhibit 7. Maltreatment Allegations for Unique Individuals with New Referrals, SFY 2015 N Unique Individuals with New Referrals Most Serious Level of Maltreatment Allegation Substantiated % 7,388 3906 52.8 Unsubstantiated 1376 18.6 Proposed 1296 17.5 Other 126 1.7 No Report 684 9.2 Note. 1. In instances of multiple reports, the report closest to the date of AFF referral was used 2. In instances when multiple allegation findings are reported, the most serious was selected 3.2 Outreach and Engagement in Services Nearly all new referrals to the AFF program (98.8%) received some form of outreach by the AFF treatment provider in their community, and the majority of these (92.4% of all new referrals) reportedly received an outreach attempt within one day or less. Exhibit 8. Outreach, SFY 2015 N New Referrals % 8562 100.0 With outreach attempt 8459 98.8 With a first outreach attempt within one (1) business day from referral 7912 92.4 537 6.3 With a first outreach attempt greater than one (>1) business day from referral Note. 1. Unit of analysis is referrals 2. Denominator used to calculate percentages is total new referrals 3. There were 10 referrals for which reported outreach date preceded referral date 4. New referrals indicate referrals that occurred in SFY 2015 Center for Applied Behavioral Health Policy Arizona State University | 12 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Once outreach has occurred, providers attempt to engage the client into services, as reflected by the client signing a release of information (ROI). This authorizes the treatment provider to gain access to clinical records of the client and to share their treatment information with DCS. In SFY 2015, just over one half (50.9%) of all referrals to the AFF program were reported to have provided a release of information and to have agreed to participate in AFF services. On average, this release of information was obtained three weeks (21 days median) after the provider had received the referral from DCS. Exhibit 9. AFF Acceptance of Services, SFY 2015 N New Referrals % 8562 100.0 With acceptance of services (with a signed ROI) 4358 50.9 Without acceptance of services (no signed ROI) 4204 49.1 Median working days from referral to ROI 21 Note. 1. Unit of analysis is referrals 2. Denominator used to calculate percentages is total new referrals 3. Median working days from referral to release of information (ROI) is business days 3.3 Patterns of Substance Abuse among Individuals Referred for Services Once an individual agrees to participate in AFF services, one of the next steps that the AFF provider will take before developing a treatment plan is to conduct a comprehensive psychosocial assessment, including an assessment of substance abuse patterns. AFF program policies allow AFF treatment providers to make use of substance abuse assessments by other providers or systems, so long as those assessments occurred within the six month period immediately preceding the referral for AFF services. Slightly more than one-half (59.7%, n=4,409) of all individuals who were referred for AFF services in SFY 2015 were assessed for substance abuse. The majority of these assessments were paid for by RBHA (60.9%), with 29.5% paid for by DCS. Center for Applied Behavioral Health Policy Arizona State University | 13 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 10. Unique Individuals with Substance Abuse Assessment and Funding Source, SFY 2015 N New Unique Individuals with Referrals in SFY 2015 % 7388 Unique Individuals without an Assessment 2979 40.3 Unique Individuals with an Assessment 4409 59.7 1297 29.5 375 8.5 2679 60.9 Private Insurance 27 0.6 Tribal 10 0.2 Medicare 9 0.2 Veteran 1 0.0 4398 100.0 Assessment Funding Source DCS only Funded Assessments Individuals with Assessments Funded by both DCS and RBHA RBHA only Funded Assessments Total Note. 1. Unit of analysis is new unique individuals. The 7,388 new unique individuals identified in this exhibit represent the 8,562 new referrals reported in Exhibit 8 2. Two data sources were used (a) AFF assessment data and (b) DHS demographic data 3. Among 4,409 unique individuals who had a recorded assessment, 11 were not included because their funding source code was zero (1) or blank (10) Assessment of substance abuse identifies an individual’s primary, secondary, and tertiary substances of abuse. Differentiating these patterns is helpful in developing a treatment plan. In SFY 2015, patterns of substance abuse were reported for a total of 3,753 individuals (85.1% of those unique individuals with a reported assessment) despite the fact that 4,409 were reported to have been assessed. Among these individuals, marijuana (52.7%), methamphetamine (47.6%), and alcohol (38%) continued to be the most commonly reported substances of abuse among individuals referred for AFF services. A comparison of primary, secondary, and tertiary patterns of substance abuse did not reveal any significant variations. Slightly more than one in five individuals (21.5%) referred to the AFF program identified heroin/opiates (13%) or other opiates/sedatives (8.7%) as their primary substance of abuse. Center for Applied Behavioral Health Policy Arizona State University | 14 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 11. AFF Patterns of Self-Reported Substance Abuse, SFY 2015 Type of Substance Use All Substances 38.0 1372 36.6 310 12.8 330 13.6 67 1.8 59 1.6 4 0.2 8 0.3 Cocaine/Crack 455 12.1 427 11.4 71 2.9 89 3.7 Hallucinogens 49 1.3 33 0.9 6 0.2 22 0.9 Heroin/Opiates 506 13.5 489 13.0 209 8.6 216 8.9 10 0.3 6 0.2 1 0.0 5 0.2 Marijuana/Hashish 1977 52.7 1915 51.1 673 27.7 674 27.9 Methamphetamine/Stimulants 1785 47.6 1746 46.6 757 31.2 752 31.1 None 364 9.7 364 9.7 364 15.0 364 15.0 Other Drugs 143 3.8 143 3.8 10 0.4 10 0.4 Other Opiates/Synthetics 326 8.7 317 8.5 80 3.3 80 3.3 7 0.2 7 0.2 1 0.0 1 0.0 18 0.5 18 0.5 5 0.2 5 0.2 Subtotal of Individuals w/SA Record 3753 85.1 3750 99.9 2430 64.7 2419 64.5 Missing Substance Abuse Records 656 14.9 Inhalants Other Sedatives/Tranquilizers Other Stimulants Total Individuals Assessed Individuals Reporting Substance Use in Preceding 30 days 1425 N % Tertiary % Benzodiazepines % Secondary N Alcohol N Primary N % 4409 3389 90.3 Note. 1. Unit of analysis is unique individuals 2. The denominator used to calculate column percentages for substances (“Total,” “Primary,” “Secondary,” “Tertiary”) is the subtotal of unique individuals with a substance abuse record 3. The denominator used to calculate row percentages for the subtotal of unique individuals with a substance abuse record (“Primary,” “Secondary,” “Tertiary”) is the subtotal of individuals with a substance abuse record (“All Substances”) 4. The denominator used to calculate “Individuals Reporting Substance Use in Preceding 30 days” is the “Subtotal of Individuals w/SA Record” minus individuals reporting “None” 3.4 Patterns of Service Delivery In SFY 2015, 7,721 new and continuing unique individuals received AFF services. Among these unique individuals who received AFF services, two thirds (63.2%) had been referred to the program during SFY 2015, while an additional 2,238 individuals (29%) had been referred in SFY 2014 but continued to receive services for some period of time during the current fiscal year. A small proportion (7.8%) of individuals with reported AFF services in SFY 2015 had been closed and then re-referred for AFF services during this year. Center for Applied Behavioral Health Policy Arizona State University | 15 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 12. Total Unique Individuals Referred and Served, SFY 2015 N % New and Continuing Referrals New and Continuing Referrals with Service(s) 12188 9004 73.9 New and Continuing Unique Individuals Referred New Individuals Both New and Continuing Continuing Individuals 10330 6706 682 2942 100.0 64.9 6.6 28.5 7721 4877 606 2238 100.0 63.2 7.8 29.0 New and Continuing Unique Individuals Served New Individuals Served Both New and Continuing Continuing Individuals Served Note. “Both New and Continuing” refers to referrals, or to unique individuals who had more than one active referral to the AFF program during the SFY, separated by one or more closures Services provided to AFF clients were jointly funded by both DCS and RBHA for nearly one half (45%) of all clients, with 32.8% of AFF clients receiving all of their services funded solely by RBHA, and 22.2% of AFF clients receiving all of their services funded exclusively by DCS. Slightly more than three quarters (77.8%) of AFF clients had some services funded by RBHA (exclusively or shared) and 67.2% had some services funded by DCS (exclusively or shared). Exhibit 13. Total Unique Individuals Served and Funding Source, SFY 2015 New and Continuing Unique Individuals Served by Funding Source DCS only Funded Individuals Individuals Funded by both DCS and RBHA RBHA only Funded Individuals N 7665 1704 3447 2514 % 100.0 22.2 45.0 32.8 New and Continuing Unique Individuals Funded by DCS and RBHA DCS-Funded Individuals RBHA-Funded Individuals 5151 5961 67.2 77.8 Note. Two providers failed to provide accurate funding information on 56 individuals with reported services. This accounts for the variance between the 7,665 “New and Continuing Individuals Served by Funding Source” and the 7,721 “New and Continuing Individuals Served” from Exhibit 12 Among those individuals with AFF services that were funded by DCS (n=5,151), “Other” represented the most commonly reported AFF Service (96.4%) and the most commonly reported substance abuse treatment service (98.9%). Among those AFF treatment services that were reported by category, mental health services (33.7%) and basic life need services (27.6%) were the more commonly reported service categories. For those AFF substance abuse Center for Applied Behavioral Health Policy Arizona State University | 16 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 treatment services that were reported by category, individual counseling (31%) and group counseling (19%) were the more commonly reported DCS-funded services. Family counseling (2.6%) and couples counseling (0%) were rarely reported. Exhibit 14. Individuals Receiving DCS-Funded Services, SFY 2015 N DCS-Funded Individuals % 5151 AFF Services Mental Health Services 1735 33.7 Basic Life Needs 1421 27.6 Medical Services 15 0.3 3 0.1 12 0.2 0 0.0 18 0.3 4967 96.4 Crisis Services Parenting Domestic Violence Job Readiness/Employment Other Substance Abuse Treatment Services Individual Counseling N % 1595 31.0 Group Counseling 981 19.0 Family Counseling 132 2.6 Couples Counseling 0.0 0.0 Substance Abuse Awareness 543 10.5 5094 98.9 Other Note. 1. These data do not capture RBHA-funded services or other services that clients may have been accessing 2. One provider failed to include in their data submissions flags to indicate funding source. For this one provider, all services were attributed to DCS funding Among those individuals with AFF services funded by RBHA (n=5,961), the more commonly reported services were support (95.4%) and treatment (87.1%). Approximately one third of individuals with RBHA-funded AFF services received medical services that could have included medication-assisted treatment for their substance abuse. A small proportion of RBHA-funded AFF clients received inpatient (6.7%) or residential treatment (7.1%) services. Center for Applied Behavioral Health Policy Arizona State University | 17 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 15. Individuals Receiving DHS-Funded Services, SFY 2015 N DHS-Funded Individuals % 5961 Service Domain Treatment Services 5191 87.1 Rehabilitation Services 1808 30.3 Medical Services 2013 33.8 Support Services 5684 95.4 Crisis Intervention Services 905 15.2 In-patient Services 398 6.7 Residential Services 426 7.1 47 0.8 Behavioral Health Day Programs Note. 1. These data do not capture other services funded by DCS or other systems to which clients may have had access 2. Two percent of all DHS encounters, representing 1,410 unique individuals, had no associated encounter code. These encounters were not included in these analyses AFF program policies require AFF providers to report levels of care for AFF clients throughout the course of their treatment. In SFY 2015, nearly two-thirds (61.7%) of all clients with AFF services at least one reported level of care assignment during their AFF treatment. Outpatient (77.4%) and intensive outpatient (31.5%) were the more commonly reported levels of care among those individuals for whom level of care was reported. Aftercare/recovery maintenance was infrequently reported (16.3%). Center for Applied Behavioral Health Policy Arizona State University | 18 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 16. Level of Care, SFY 2015 N % New and Continuing Unique Individuals Served 7721 100.0 New and Continuing Unique Individuals with at Least One Level of Care Identified 4760 61.7 Outpatient 3685 77.4 Intensive Outpatient 1499 31.5 Residential – Adult 29 0.6 Residential – Child 0 0.0 778 16.3 2961 38.3 Recovery Maintenance/Aftercare New and Continuing Individuals with No Level of Care Identified Note. 1. LOC categories are not mutually exclusive. Individuals can be assigned to multiple levels of care throughout their AFF program experience 2. Individuals assigned to the “No Level of Care Identified” had no record of LOC assignment at any time during their AFF program experience 3.5 Patterns of Program Closure and Length of Service A total of 7,486 closures were reported by AFF providers from the 12,188 new and continuing referrals for AFF services that were processed in SFY 2015. The median duration of AFF program participation among all reported closures was 98 days (mean=143.4, SD=126.4). Closure categories are grouped into four primary categories: “unable to locate/refused services”; “discontinued services after AFF services had begun”; “completed AFF services”; and “other.” The category of “unable to locate/refused services” represented the single largest category of reported closures, accounting for 4,954 (67%) of all reported closures. These closures occurred soon after a DCS employee had made a referral for services, during the period that the provider was attempting outreach, engagement, and assessment activities. “Discontinued” represents the most commonly reported closure reason among referrals that had been engaged in AFF treatment services, representing 1,622 referrals or 21.7% of all reported closures. The median duration of AFF program participation among these referrals was nearly six months (median= 176, mean=203). Completion of AFF treatment services was reported for just 864 of the 7,486 reported closures, representing 11.54% of all reported closures. The median duration of AFF participation among these referrals was 335 days (mean=312.8). An additional 31 closures (.41%) were reported as completed AFF services following aftercare/recovery maintenance. Program participation for these referrals was of comparable duration to the previous group, with a reported median of 306 days. Center for Applied Behavioral Health Policy Arizona State University | 19 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 17. AFF Closures and Length of AFF Participation, SFY 2015 N Completed DisContinued Unable to Locate/ Refused Other % Total New and Continuing Referrals 12188 Closures Completed AFF At The Conclusion Of SA Treatment Completed AFF At The Conclusion Of Recovery Maintenance Client Discontinued W/O Completing Services (Excluding Unable To Locate) Unable To Locate For Initial Outreach 7486 Unable To Locate For Intake Median Mean Standard Deviation Length of Participation in Days 98 143.4 126.4 864 11.5 335 312.8 147.8 31 0.4 306 297.1 86.3 1622 21.7 176 203.1 114.7 1721 23.0 47 63.9 58.9 1499 20.0 82 101.6 72.3 Unable To Locate (Post-Intake) 332 4.4 166.5 192.5 116.6 No SA Problem Client Refused Service At Initial Referral Or Assessment Incarcerated 647 8.6 76 96.7 76.9 515 6.9 53 77.9 88.6 155 2.1 92 138.0 126.0 Move Out Of Area 85 1.1 130 142.6 106.1 Death 15 0.2 130 146.5 170.9 Note. 1. Unit of analysis is referrals 2. AFF participation is measured from the date of referral to the date of closure 3. Five referrals had closures dates before their referral data and were excluded from analysis Center for Applied Behavioral Health Policy Arizona State University | 20 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 4.0 4.1 Program Outcomes Patterns and Results of Drug Testing Slightly more than one half (51%) of all new and continuing individuals were referred at least once for a drug test. Among these 5,270 individuals, a total of 64,238 drug test referrals were reported. The monthly rate of drug referrals was calculated for those individuals who had 30 days or more or AFF service participation (n=3,365). Among these individuals, the average number of drug test referrals per month of AFF participation was 2.8 (SD=2.5) with a median rate of 2.02 drug test referrals per month. Nearly four fifths (79.1%) of drug test referrals resulted in a test result, while one fifth (20.7%) of the drug test referrals clients failed to appear for the test. Less than 1 in 6 (14.1%) test results detected the presence of drugs, while nearly 85% (84.9%) of all usable test results did not detect the presence of any illicit substances. Exhibit 18. Patterns and Results of Drug Testing, SFY 2015 N % New and Continuing Unique Individuals Served 7721 100.0 Individuals with Drug Test Referrals 5081 65.8 Individuals without Drug Test Referrals 2640 34.2 Total Number of Drug Test Referrals 64238 Client Failed to Appear for Test 13322 20.7 Client Refused 36 0.1 Drug Test Cancelled for Reasons Beyond Client Control 47 0.1 Drug Test Referrals with Results 50833 79.1 Total Number of Drug Test Results 50833 Positive Test Results 7152 14.1 Negative Test Results 43172 84.9 427 0.1 Altered Specimen/Sample 10 0.0 Test Indicates Allowable Substance 72 0.0 Awaiting Results Note. 1. Drug referrals per month were computed for 3,365 unique individuals who had 30 days or more of AFF service participation 2. The average frequency of drug test referrals was 2.8 (SD=2.5) with a median 2.02 per month of AFF participation 4.2 Employment Center for Applied Behavioral Health Policy Arizona State University | 21 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Comparative analysis of employment status at the time of AFF referral and at the time of AFF program closure was conducted for 2,229 referrals (29.8% of all recorded AFF closures) that were closed in the current state fiscal year and for which employment status information was available at intake and at closure. Closed referrals were not included for 5,257 cases (70.2% of all recorded AFF closures) because employment status information was not included by the provider in the intake and/or closure record. Referrals that would subsequently close as Completed AFF Services demonstrated higher rates of full- and part-time employment at intake, and correspondingly lower unemployment rates, than referrals that would subsequently close as Discontinued AFF Services. Furthermore, referrals closed as Completed AFF Services demonstrated greater gains in employment at closure, relative to intake, along with corresponding reduction in unemployment, in comparison to Discontinued AFF Services closed referrals. However, both Completed and Discontinued referrals demonstrated gains in full-time employment with corresponding reductions in unemployment at closure relative to intake. Exhibit 19. Employment Outcomes, SFY 2015 Completed AFF Services N = 799 Intake N Employed, Full-time Employed, Part-time Subtotal Discontinued AFF Services N = 1430 Closure % N Intake % N Closure % N % 202 25.3 293 36.7* 244 17.1 311 21.7* 119 14.9 126 15.8 165 11.5 156 10.9 321 40.2 419 52.4* 409 28.6 467 32.7* Unemployed 300 37.5 203 25.4* 800 55.9 749 52.4 Other Unknown 40 138 799 5.0 17.3 47 130 799 5.9 16.3 75 146 1430 5.2 10.2 77 137 1430 5.4 9.6 Total Note. 1. Unit of analysis is new and continuing individuals 2. “Intake” indicates employment status at assessment and “Closure” indicates employment status at closure 3. The employment category "Other" inside each closure group collectively refers to individuals who fall in one of the following employment categories: volunteer, unpaid rehabilitation, homemaker, student, retired, disabled, inmate of institution, and transitional employment placement * Status at “Closure” significantly different than status at” Intake,” p < .001 Center for Applied Behavioral Health Policy Arizona State University | 22 Arizona Families F.I.R.S.T. 4.3 Annual Evaluation Report SFY 2015 Longitudinal Analysis of Permanency A total of 6,383 children, associated with 2,828 parents who were served and were closed in the AFF program in SFY 2014, were placed in out-of-home-care. Of these, 54% were associated with parents who could not be located or had refused AFF services. Among the 3,066 children who achieved permanency at follow up, those whose parents had successfully completed their AFF program were significantly more likely to experience reunification (84.8%) than children whose parents had discontinued (52.6%), could not be located or refused AFF services (45.1%), or other (41.9%). Children whose parents were closed as “other” (included cases of death and imprisonment) experienced the highest rate of guardianship (22.3%), while children whose parents successfully completed the AFF program experienced the lowest rate of adoption (9.5%). However, these results should be interpreted with caution since adoption may take upwards of two years and program participation may not be the only requirement for reunification. Exhibit 20. Permanency and Reunification, SFY 2014 Cohort Total Adults Total Children N % N Completed 458 16.2 Discontinued Unable to Locate/Refused Other 709 Total Permanency Still in Care % N 1,067 16.7 699 65.5* 351 32.9* 17 1.6 25.1 1,621 25.4 648 40.0 944 58.2 29 1.8 1,556 55.0 3,448 54.0 1,607 46.6 1,760 51.0 81 2.3 105 3.7 247 3.9 112 45.3 131 53.0 4 1.6 2,828 100.0 6,383 100.0 3,066 48.0 3,186 49.9 131 2.1 Total Children % N Permanency Outcome Reunification Guardianship N % N % N % Completed 699 22.8 593 84.8* 39 Discontinued Unable to Locate/Refused Other 648 21.1 341 52.6 1,607 52.4 726 112 3.7 3,066 100.0 Total Other % N % Adoption N % 5.5** 67 9.5* 61 9.4 246 37.9 45.1 116 7.2 765 47.6 47 41.9 25 22.3 40 35.7 1,707 55.6 241 7.8 1,118 36.4 Note. 1. In cases of multiple referrals for the same individual, the most recent referral in SFY 2014 was used 2. In cases of multiple removals for the same child, the most recent removal in SFY 2014 was used * “Completed” significantly different than “Discontinued,” “Unable to Locate/Refused,” and “Other,” p < .01 Center for Applied Behavioral Health Policy Arizona State University | 23 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 ** “Completed” significantly different than “Other,” p < .01 4.4 Longitudinal Analysis of Maltreatment Recidivism A total of 5,075 new and continuing SFY 2014 referrals for which usable maltreatment allegation data were provided were observed in SFY 2015. Slightly more than two thirds (67.8%) were associated with at least one substantiated allegation of maltreatment at the time of their referral to the AFF program. An additional 18.9% of all referrals in SFY 2014 were associated with at least one allegation of unsubstantiated maltreatment at the time of AFF referral. Rates of substantiated allegations at the time of AFF referral between the four closure groups were as follows: completed (67.9%), discontinued (70.0%), could not locate (66.5%), and other (71.8%). A statistical comparison of the rates of No Report at follow-up with substantiated reports at the time of AFF referral demonstrates that for all groups (completed, discontinued, refused/could not be located, other) the maltreatment recurrence is significantly reduced. The percentages of substantiated allegation cases at AFF referral that went on to have no follow-up allegation were 91.9% (completed), 81.1% (discontinued), 87.7% (could not locate), and 90.9% (other closures), demonstrating that high rates of No Report are associated with program referrals, regardless of whether treatment was completed. Center for Applied Behavioral Health Policy Arizona State University | 24 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Exhibit 21. Maltreatment Recurrence, SFY 2014 Cohort Longitudinal Analysis Unable to Locate/ Refused* Discontinued* Completed* Pre-Referral Allegation Findings Unsubst. Proposed No Report % N % N N % N % N % 587 67.9 26 4.4 75 12.8 14 2.4 12 2.0 460 78.4 Unsubst. 167 19.3 4 2.4 33 19.8 4 2.4 9 5.4 117 70.1 Proposed 22 2.5 0 0.0 2 9.1 1 4.5 1 4.5 18 81.8 2 0.2 0 0.0 0 0.0 0 0.0 0 0.0 2 100.0 Other No Report % Other N 86 9.9 2 2.3 3 3.5 1 1.2 1 1.2 79 91.9 Total 864 100 32 3.7 113 13.1 20 2.3 23 2.7 676 78.2 Subst. 875 70.1 70 8.0 70 8.0 40 4.6 19 2.2 676 77.3 Unsubst. 222 17.7 20 9.0 36 16.2 10 4.5 12 5.4 144 64.9 Proposed 42 3.3 3 7.1 3 7.1 1 2.4 1 2.4 34 81.0 3 0.2 0 0.0 0 0.0 0 0.0 1 33.3 2 66.7 Other No Report 106 8.4 4 3.8 10 9.4 5 4.7 1 0.9 86 81.1 Total 1,248 100 97 7.8 119 9.5 56 4.5 34 2.7 942 75.5 Subst. 1,848 66.5 125 6.8 154 8.3 63 3.4 54 2.9 1,452 78.6 Unsubst. 540 19.4 42 7.8 89 16.5 20 3.7 27 5.0 362 67.0 Proposed 99 3.5 8 8.1 3 3.0 1 1.0 2 2.0 85 85.9 Other 12 0.4 0 0.0 0 0.0 0 0.0 5 41.7 7 58.3 No Report Subst. Other* Subst. Subst. Total 276 9.9 8 2.9 17 6.2 6 2.2 3 1.1 242 87.7 2,775 100.0 183 6.6 263 9.5 90 3.2 91 3.3 2,148 77.4 135 71.8 7 5.2 5 3.7 1 0.7 2 1.5 120 88.9 Unsubst. 34 18.0 3 8.8 5 14.7 0 0.0 1 2.9 25 73.5 Proposed 8 4.2 1 12.5 2 25.0 0 0.0 1 12.5 4 50.0 Other 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 No Report Total Subst. Total* Post-Referral Allegation Findings 11 5.8 0 0.0 0 0.0 0 0.0 1 9.1 10 90.9 188 100 11 5.9 12 6.4 1 0.5 5 2.7 159 84.6 3,445 67.8 228 6.6 304 8.8 118 3.4 87 2.5 2,708 78.6 Unsubst. 963 18.9 69 7.2 163 16.9 34 3.5 49 5.1 648 67.3 Proposed 171 3.3 12 7.0 10 5.8 3 1.8 5 2.9 141 82.5 17 0.3 0 0.0 0 0.0 0 0.0 6 35.3 11 64.7 Other No Report Total 479 9.4 14 2.9 30 6.3 12 2.5 6 1.3 417 87.1 5,075 100 323 6.4 507 10.0 167 3.3 153 3.0 3,925 77.3 Note. 1. Subst. = Substantiated; Unsubst. = Unsubstantiated 2. Most recent 2014 referral (that is closed) for each client chosen as the base referral 3. Most recent allegation prior to or on the same day as the referral (1) is chosen as the pre-referral allegation; closest allegation to the referral date occurring before the referral date 4. Earliest allegation after the referral date (1) is chosen as the post-referral allegation; closest allegation report to the referral date after the referral date 5. In situations when maltreatment reports had multiple allegation findings the most serious allegation was selected * Post-referral percent of referrals with No Report allegation is significantly different than pre-referral percent of referrals with No Report allegation, p < .001 Center for Applied Behavioral Health Policy Arizona State University | 25 Arizona Families F.I.R.S.T. Center for Applied Behavioral Health Policy Annual Evaluation Report SFY 2015 Arizona State University | 26 Arizona Families F.I.R.S.T. 5.0 Annual Evaluation Report SFY 2015 Key Findings and Program Implications This report summarizes the key processes and outcomes of the Arizona Families F.I.R.S.T. program during SFY 2015 (July 1, 2014 – June 30, 2015). As a legislatively mandated element of the AFF program, this annual evaluation report provides analysis of the performance of DCS and its contracted AFF providers in meeting the legislative mandates of the program. These mandates include: 1. Increasing the availability, timeliness and accessibility of substance abuse treatment to improve child safety, family stability, and permanency for children in foster care or other out-of-home placement, with a preference for reunification with a child's birth family; 2. Increasing the availability, timeliness and accessibility of substance abuse treatment to persons receiving temporary assistance for needy families to achieve self-sufficiency through employment; and 3. Increasing the availability, timeliness and accessibility of substance abuse treatment to promote recovery from alcohol and drug problems. The evaluation of the AFF program and the preparation of this report are conducted independently by the Arizona State University Center for Applied Behavioral Health Policy. The content of this report, and the conclusions contained herein, represent the opinions of ASU and do not necessarily reflect the opinions of, or endorsement by, DCS. A number of changes in programmatic requirements and associated reporting requirements were implemented this year, limiting the validity of comparisons between performance and outcomes this year and preceding years. Readers are cautioned in making such comparisons except where noted in this report. Additionally, this year’s report introduces for the first time a longitudinal analysis of the impact of the AFF program upon family reunification and child safety (maltreatment recurrence). In the following sections, the key findings are summarized for each of the legislatively mandated outcomes and service processes, followed by program implications of these findings. 5.1 Child Permanency and Reunification Beginning with this year’s report, child permanency, reunification, and safety were analyzed among those families that had concluded their AFF services in SFY 2014. The movement to this longitudinal analysis of these two outcomes was in response to the recognition among DCS officials and ASU researchers, along with peer-reviewed research, that estimation of permanency and maltreatment recurrence within the same year of program participation underestimated the true rates of these events. Forty-nine percent (49%) of the children associated with parents who had received AFF services in SFY were still in care at the conclusion of SFY 2015. Forty-eight percent (48%) of Center for Applied Behavioral Health Policy Arizona State University | 27 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 the children associated with parents who had received AFF services during SFY 2014 had achieved permanency by the conclusion of SFY 2015. Children whose parents had completed their AFF program in SFY 2014 were significantly more likely to achieve permanency that those whose parents had discontinued their AFF services. Conversely, children who were still in care at the conclusion of 2015 were more likely to be associated with parents who had discontinued their participation in AFF services than with parents who had completed their program. Fifty-five percent (55.6%) of those children who had achieved permanency were reunified with their families, while 36.4% of children who achieved permanency were adopted. Children whose parents had completed the AFF program were significantly more likely to experience reunification than children whose parents had discontinued AFF program (84.8% vs. 52.6%). Likewise, children whose parents had discontinued their AFF program were significantly more likely to be adopted than children whose parents had completed the AFF program (37.9% vs. 9.5%) These findings provide clear and compelling evidence of the effectiveness of the AFF program in achieving the legislative goal of improving family stability and permanency for children in foster care or other out-of-home placement, with a preference for reunification with a child's birth family. Furthermore, these findings suggest that the AFF program produces reunification results that are superior to the DCS program in general or to nationwide trends. According to a recently released report from the Arizona Auditor General Office, 6 the overall rate of reunification within Arizona was about 33%, in comparison to 40% nationally. The observed finding that among families that complete the AFF program, permanency occurs for 65.5% of the children, with family reunification occurring in 84.8% of the cases, provide compelling evidence of the impact of the AFF program. 5.2 Child Safety Among the 5,075 referrals that were reported for SFY 2014, approximately 90% had one or more allegation of child maltreatment at the time of their referral to the AFF program. Sixty-eight percent (67.8%) of the referrals had at least one substantiated allegation at the time of referral, with an additional 18.9% having at least one unsubstantiated allegation at the time of referral. At the conclusion of SFY 2015, 22.7% of these referrals had been accused of a subsequent act of child maltreatment. Parents who had at least one unsubstantiated claim of maltreatment at the time of AFF referral had the highest rates of re-allegation (32.7%), while parents with an proposed allegation at AFF referral had the lowest rate of re-allegation (17.5%). Parental completion or discontinuation of AFF program participation in SFY 2014 provide weak evidence of affecting the likelihood of maltreatment re-allegation. Parents who completed their AFF program participation in SFY 2014 had a re-allegation rate of 21.8%, in comparison to a re6 Arizona Office of the Auditor General. (2015). Arizona Department of Child Safety Independent Review: Supplemental Report: Background Data. Center for Applied Behavioral Health Policy Arizona State University | 28 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 allegation rate of 24.5% for parents who had discontinued their AFF participation in SFY 2014. While promising, this distinction is statistically insignificant. Although the tests suggest that in the short window of one year following up from referral, the rate of maltreatment re-allegation is not affected by successful AFF program completion, a longer time-frame is really needed to assess the true impact of maltreatment re-allegation from program completion. Overall, these findings provide promising evidence of the effectiveness of referral, offering parents an opportunity to reflect on the impact of their substance abuse as it relates to child maltreatment allegation. Some recent research on screening and brief interventions in substance abuse suggests that such brief intervention can be very effective. 7 At least for the time frame of one year, the referral to the program appears to support the legislative goal of improving child safety. With a longer cohort analysis, the actual impact of program completion can be more readily determined. However, in the interim, providers who can utilize brief intervention techniques at outreach and engagement may be providing a significant intervention to reduce child maltreatment re-allegation. 5.3 Parental Substance Abuse At the time of AFF referral, 85.1% of parents were assessed for substance abuse in SFY 2015. Unfortunately, 14.9% of all referrals had no record of substance abuse assessment at the time of their referral. Nonetheless, among those parents who had a recorded substance abuse assessment, cannabis (52.7%), methamphetamine (47.9%), and alcohol (38%) were the more commonly reported substances of abuse. Consistent with national trends, AFF clients selfreported the use of heroin and other opiates at an alarming and increasing rate (22.2%). Drug testing is an integral element in the AFF program model, and AFF providers are expected to refer individuals for drug testing and to report the results of these tests on a routine basis. In SFY 2015, AFF providers reported referring 51% of their clients for drug tests. Unfortunately, 49% of all new and continuing individuals served in SFY 2015 had no reported drug test referral. A total of 64,238 drug test referrals were reported in SFY 2015, representing 5,270 new and continuing AFF clients. Clients failed to appear for their drug tests in 20.1% of the situations, with 79.1% of all drug test referrals resulting in usable drug test results. The overwhelming majority of these tests (84.9%) failed to detect any drug use, suggesting abstinence. In just 14.1% of the drug tests was drug use detected; provider reporting specifications do not allow for an analysis of the detected drugs. Assessment of parents’ drug use at the time of their referral, along with the results of their drug tests conducted during their AFF program participation, provide strong evidence of the effectiveness of the AFF program in achieving the legislative goal of promoting recovery from alcohol and drug problems. Nearly all (99.1%) individuals who were assessed for substance abuse at the time of their AFF referral reported problems with alcohol or drugs. Eight-five 7 McCambridge, J., Witton, J., & Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. Journal of clinical epidemiology, 67(3), 267-277 Center for Applied Behavioral Health Policy Arizona State University | 29 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 percent (85%) of all drug tests conducted during an individual’s participation in the AFF program indicated no drug use. These results provide strong and compelling evidence of the effectiveness of the AFF program in achieving the legislative goal of promoting recovery by reducing drug and alcohol use during AFF program participation. Unfortunately, the lack of any follow-up assessment of participants following AFF participation limited the ability to evaluate the impact of the AFF program in promoting long-term recovery in a manner comparable to that reported for permanency and child safety. 5.4 Parental Employment At the time of referral to the AFF program, 31.6% of parents reported being employed, with 20.8% reporting full-time employment and 10.8% reporting part-time employment. An additional 39.6% reported being unemployed at the time of AFF referral. Unfortunately, unemployment status was not reported to ASU for 23.7% of the referrals. Individuals who completed AFF services were more likely to report employment at the time of their AFF closure than clients who discontinued AFF services (52.5% vs. 32.6%). Conversely, individuals who discontinued AFF services were more likely to report being unemployed at the time of their AFF closure than individuals who had completed the AFF program (52.4% vs. 25.4%). Interestingly, individuals who completed the AFF program reported higher rates of employment at the time of their AFF referral when compared to those individuals who discontinued AFF services (40.2% vs. 28.6%). Nonetheless, the gains in employment status among those completing the AFF program were still significantly greater than the gains in employment among those discontinuing the AFF program, after controlling for the differential rates of employment at intake. These results provide strong and compelling evidence of the effectiveness of the AFF program in achieving the legislative goal of promoting self-sufficiency through employment. Unfortunately, limitations in the data from the TANF and Jobs programs do not allow for an adequate assessment of the effectiveness of the AFF program in proving services to persons receiving TANF. 5.5 Availability, Timeliness, and Accessibility of Substance Abuse Treatment Services The number of referrals to the AFF program in SFY 2015 was 8,562 and represented a 34% increase over SFY 2014. This rise in rate of referrals parallels a number of other indicators in the DCS program, including the rate of out-of-home removals from 2009 – 2013 of 36%. 8 AFF providers reported providing timely outreach to these referrals, with 92.4% of referrals reportedly receiving outreach within one day or less. However, just over one-half (50.9%) of all referrals resulted in the client providing a release of information, signifying their acceptance of AFF services. On average, this release of information wasn’t obtained until 21 days after the referral had been received by the treatment provider. Among those individuals who did not provide a 8 Arizona Office of the Auditor General. (2015). Arizona Department of Child Safety Independent Review: Supplemental Report: Background Data. Center for Applied Behavioral Health Policy Arizona State University | 30 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 release and information, and for whom the provider subsequently filed a closure report, the inability of the provider to locate the client for initial outreach (23%) or for intake (20%) were frequent occurrences. Client refusal to accept services was infrequently cited by the AFF provider as a reason for case closure (6.9%). AFF services were provided in SFY 2015 to 4,877 new individuals (out of 6706 new individuals who had been referred). As such, these results provide promising evidence of the effectiveness of the AFF program in meeting the legislative goal of Increasing the availability, timeliness, and accessibility of substance abuse treatment. On the one hand, more individuals were referred to the AFF program than in previous years, and providers reported outreaching to these referrals quite rapidly. On the other hand, just over one-half of the referrals resulted in a client accepting AFF services and this acceptance did not occur until three weeks following referral. Among those referrals that did not result in a client accepting AFF services, the inability of the provider to locate the client was a predominant reason, suggesting problems in the referral information that is received by the AFF provider from DCS. 5.6 Conclusions, Implications, and Next Steps Evidence contained within this report reflects positively upon AFF in achieving the legislatively mandated goals for the program. However, a number of trends underscore areas for continuing improvement in program services and program reporting. These are highlighted below. Transforming Referrals to Clients: More than two-fifths (43%) of all case closures were attributed to the failure of the AFF provider to locate the client. Similarly, the rate of AFF referrals resulting in a client providing a release of information and formally accepting AFF services was just 50.9%. These two data trends suggest a certain degree of inefficiency in the referral processes from DCS to the AFF providers, and/or the providers’ effectiveness in successfully engaging referred individuals into treatment. DCS may wish to consider implementing process improvement strategies to improve the referral information from DCS and/or AFF providers, to successfully engage these individuals, and secure client acceptance of services. Improving Client Success: Just over 10% of all case closures reported in SFY 2015 were classified as “Completed,” indicating that the client had successfully completed their AFF treatment program. In contrast, the rate of closures classified as “Discontinued,” indicating that the client had either dropped out or had been terminated from services, was 21.7% of all case closures. This low rate of successful program completion limits the potential impact of the AFF program, particularly in light of the positive impact that AFF program completion has upon permanency and parental employment. DCS may wish to consider implementing process improvement strategies to improve the ability of AFF providers to retain clients and to ensure their successful completion of the AFF program. Center for Applied Behavioral Health Policy Arizona State University | 31 Arizona Families F.I.R.S.T. Annual Evaluation Report SFY 2015 Improving AFF Provider Documentation: During SFY 2015, ASU independently implemented a series of enhancements to the data collection processes in an effort to enhance the accuracy of the information submitted to ASU from the AFF providers. These enhancements, which were not part of ASU’s contracted scope of work, made significant improvements in the quality of data that it received from the providers. Nonetheless, there remain significant areas for improvement in these processes, as evidenced by a number of suspicious patterns in the data provided to ASU. For example, it is not clear if 49.1% of all referrals did not accept services or if AFF providers simply failed to report these occurrence. While only 4,358 referrals were reported to have accepted services, 4,788 new individuals were reported to have received services, suggesting some inconsistency in provider reporting. AFF providers are expected to report DCS-funded services in eight AFF service categories and six substance abuse treatment categories. In both instances however, the category “other” is used for nearly 100% of the clients (96.4% for AFF services and 98.9% for substance abuse services), more than three times the rates of any other category. Similarly, AFF providers are supposed to report clients’ levels of care throughout their AFF program participation. However, more than one half of all clients that received AFF services in SFY 2015 had no reported level of care. In the upcoming year, and in coordination with DCS, ASU will continue to enhance its data validation procedures and to implement specific process improvement measures to increase the quality and information submitted by the AFF providers, thereby enhancing DCS’ ability to more accurately report on the key service delivery and service outcome indicators of this important and effective program. Center for Applied Behavioral Health Policy Arizona State University | 32