Arizona Families F.I.R.S.T. Program Annual Evaluation Report for the Period July 1, 2003 – June 30, 2004 Submitted on: February 28, 2005 Submitted to: Arizona Department of Economic Security Division of Children, Youth and Facilities Phoenix, Arizona Contract #: Submitted by: E4374163 Applied Behavioral Health Policy The University of Arizona 721 N. Fourth Avenue, Suite 107 Tucson, AZ 85705 (520) 917-0841 Fax (520) 917-0845 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 ACKNOWLEDGEMENTS This report was prepared by Applied Behavioral Health Policy (ABHP), The University of Arizona, under contract number E4374163 with the Arizona Department of Economic Security, Division of Children, Youth and Families (ADES/DCYF), in partnership with the Arizona Department of Health Services, Division of Behavioral Health Services (ADHS/DBHS) through the Joint Substance Abuse Treatment Fund. At ABHP, Brian L. Arthur was responsible for the content, analysis, and writing of the report, and Michael S. Shafer, Ph.D., reviewed and provided numerous suggestions and improvements on its content and format. The following ABHP staff were responsible for other support and analysis: George Bentley, Jim Bradley, Lucinda Davis, Heather Free, Gene Grzywacz, John Hannah, Ahmet Kucuk, Manjula Mellachuruvu, Megan Multanen and Ken O’Day. The authors wish to thank staff of the Arizona Department of Economic Security/Division of Children, Youth and Families, and the Arizona Department of Health Services/Division of Behavioral Health Services, for their ongoing cooperation and assistance with the evaluation. In particular, Steve Sparks, Nancy Hansen and Carmen Preciado of ADES, and Christy Dye and Cora Bagley of ADHS have been very helpful throughout the course of the evaluation. Thanks are extended as well to the staff of Jaimie Leopold and Associates for their work in conducting AFF provider stakeholder interviews and client focus groups. This report was funded through a contract with the Arizona Department of Economic Security in partnership with the Arizona Department of Health Services, through the Joint Substance Abuse Treatment Fund. Points of view represented in this report are those of the authors, and do not necessarily represent the official position or policies of either the Arizona Department of Economic Security or the Arizona Department of Health Services. Suggested citation: Applied Behavioral Health Policy. (2005). Arizona Families F.I.R.S.T. Program: Annual Evaluation Report for the Period July 1, 2003 – June 30, 2004. Tucson, AZ: The University of Arizona. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 2 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 TABLE OF CONTENTS Chapter Page Acknowledgements ........................................................................................................................2 Table of Contents ...........................................................................................................................3 List of Tables ..................................................................................................................................5 Executive Summary .......................................................................................................................6 1. Introduction......................................................................................................................11 1.1 Background Information on the Arizona Families F.I.R.S.T. Program...................11 1.2 Current Statewide Context of AFF Program Operations.......................................13 1.3 Overview of the Evaluation Framework................................................................13 1.4 Data Sources for the Annual Report ......................................................................13 1.5 Organization and Contents of the Annual Evaluation Report ...............................14 2. Description of Arizona Families F.I.R.S.T. Clients and Services Received ..................16 2.1 Referrals to the AFF Program................................................................................16 2.2 Characteristics of Individuals Referred to the AFF Program ................................18 2.3 Assessments ...........................................................................................................20 2.4 Characteristics of AFF Participating Clients .........................................................21 2.5 Substance Use Among AFF Participating Clients At Time of Enrollment ...........25 2.6 Engagement in Treatment Among Participating Clients .......................................27 2.7 Substance Abuse Treatment Services Among Participating Clients .....................27 2.8 Most Intensive Substance Abuse Treatment Services among Participating Clients...................................................................................28 2.9 Time Spent in Treatment Among Participating Clients.........................................30 3. Description of Outcomes Data ........................................................................................32 3.1 Methodology ..........................................................................................................32 3.2 Child Welfare Outcomes Among AFF Clients Referred from CPS: Is There a Recurrence of Child Abuse and/or Neglect Among CPS Families Participating in AFF? ..............................................................................32 3.3 Child Welfare Outcomes Among AFF Clients Referred from CPS: Are Children in Foster Care Whose Caregivers Enroll in AFF Reunified with Their Caregivers?..........................................................................34 3.4 Employment Outcomes Among AFF Clients: Do AFF Participants Who Were Employed Maintain Their Employment? Do AFF Participants Who Were Unemployed Obtain Employment? .....................................................35 4. Implementation Issues of the Arizona Families F.I.R.S.T. Program ............................37 4.1 Methodology ..........................................................................................................37 4.2 Timeliness of Service Delivery..............................................................................38 Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 3 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 4.3 4.4 4.5 4.6 5. Availability of Drug Treatment Services...............................................................40 Accessibility of Drug Treatment Services .............................................................41 Organizational Collaboration and Case Coordination ...........................................43 Opportunities For Enhancement ............................................................................44 Summary and Conclusions..............................................................................................47 5.1 Major Findings.......................................................................................................47 5.2 Conclusion and Recommendations........................................................................48 Appendices ................................................................................................................................50 Appendix A: Revised Evaluation Plan ..................................................................51 Appendix B: Listing of Qualitative Study Participants and Protocols ..................58 Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 4 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 TABLES Table 1.1.1 2.1.1 2.1.2 2.1.3 2.2.1 2.3.1 2.3.2 2.4.1 2.4.2 2.5.1 2.5.2 2.5.3 2.6.1 2.7.1 2.8.1 2.9.1 3.2.1 3.2.2 3.3.1 3.3.2 ............................................................................................................................Page AFF Provider Agencies, DES Districts, and Counties Served ..........................................11 AFF Program Referrals By Providers and Quarters ..........................................................15 Cumulative AFF Program Referrals ..................................................................................16 Cumulative AFF Program Referral Sources ......................................................................16 AFF Program Referral Demographic Characteristics........................................................18 AFF Program Assessments ................................................................................................19 Cumulative AFF Program Assessments ............................................................................20 AFF Participating Clients ..................................................................................................21 AFF Participating Clients’ Demographic Characteristics..................................................22 Types of Substance Used By AFF Participating Clients 30 Days Prior to Enrollment ............................................................................................................24 Poly-Drug Use Among AFF Participating Clients ............................................................25 Poly-Drug Use Patterns Among AFF Participating Clients at Time of Enrollment .....................................................................................................................25 Engagement Rates Among AFF Clients During State Fiscal Year 2004 ..........................26 Types of Services Used by AFF Participating Clients During State Fiscal Year 2004 .......................................................................................................28 AFF Participating Clients’ Most Intensive Treatment Service Modality and Secondary Services Received .....................................................................................29 Treatment Retention: Time Spent in Treatment for Clients with at Least Six Months Opportunity...........................................................................................29 Substantiated CPS Reports Among CPS Referred AFF Participating Clients ..................32 Substantiated CPS Reports: Types of Child Abuse/Neglect Among AFF Participating Families ................................................................................................33 Reunification of Children Placed in Foster Care ...............................................................33 Days in Care for Children Reunified with Parent(s) or Caregiver.....................................34 Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 5 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 EXECUTIVE SUMMARY Arizona Families F.I.R.S.T. and Its Development in Brief A rizona Families F.I.R.S.T. (AFF) was established by Arizona Revised Statute (ARS) 8-881 (Senate Bill 1280, passed in the 2000 legislative session), and is administered jointly by the Arizona Department of Economic Security (ADES) and the Arizona Department of Health Services (ADHS), with DES designated as the lead agency. The legislation established a statewide program for substance-abusing families entering the child welfare system as well as those families receiving cash assistance through Temporary Assistance for Needy Families (TANF). The legislation recognized that substance abuse is a major problem contributing to child abuse and neglect, and is also a significant barrier for those attempting to re-enter the job market or maintain employment. In the spring of 2001, nine AFF providers received contracts through ADES to implement a community substance abuse prevention and treatment program under Arizona Families F.I.R.S.T. Contract providers across the State of Arizona were funded so that all counties would be covered by AFF services. The agencies funded included: Arizona Partnership for Children-Coconino, Arizona Partnership for Children-Yavapai, and Arizona Partnership for Children-Yuma; Community Partnership of Southern Arizona; Horizon Human Services; Old Concho Community Assistance Center; Southeastern Arizona Behavioral Health Services; TERROS; and WestCare Arizona. Over the past three years of program operations, AFF provider agencies worked to: develop a referral process; screen, access, and treat clients with the required AFF timeframes; develop collaborative partnerships with subcontractors and other community agencies; and coordinate treatment services with Regional Behavioral Health Authority (RBHA) providers when the AFF client was found to be eligible for Medicaid-Title XIX funded services. Provider agencies also have worked to promote a more family-centered service delivery system, and to engage and retain clients in treatment. AFF providers and RBHA providers coordinate efforts to serve eligible clients in a manner that maximizes resources and Title XIX/XXI funds. Through the Partnership each eligible person would be afforded access to a comprehensive array of Title XIX behavioral health services that will assist, support, and encourage that person to achieve and maintain health and self-sufficiency. The evaluation of AFF, required by ARS 8-881, focuses on the implementation of the AFF community substance abuse prevention and treatment program at all nine sites, the factors that contribute to their success, and the extent to which the legislature’s outcome goals of increases in timeliness, availability and accessibility of services; recovery from alcohol and drug problems; child safety; permanency for children through reunification; and the achievement of selfsufficiency through employment can be obtained. This year’s evaluation continues to focus on the documentation of program implementation through the analysis and reporting of client-level service utilization data from AFF providers and the Department of Health Services/Division of Behavioral Health Services, and qualitative data gathered from AFF program directors, RBHA and Child Protective Services (CPS) representatives, AFF clients, and other stakeholders. Analyses also were conducted with respect to child welfare outcomes as of June 30, 2004. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 6 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Overview of the Annual Evaluation Report This report presents service utilization data for the annual reporting period that covers July 1, 2003 through June 30, 2004. The evaluation data have contributed to an understanding of the characteristics of AFF participating clients; the types of substances used by clients, including poly-drug use patterns; referral trends; levels of client engagement in services; service utilization patterns; and lengths of stay in treatment. Process data presented in this year’s Annual Evaluation Report were collected through a variety of methodologies including in-person and telephone interviews, focus groups, and document reviews. More than 200 stakeholders and clients participated in interviews or focus groups commenting on various aspects of the AFF program. Results related to treatment and recovery reported this year include the findings that AFF clients are engaged in treatment services at a high rate and are spending several months in treatment services. These are encouraging results because retention of clients in treatment services to address their needs is an intermediary outcome in the recovery process. Outcomes in the areas of child welfare provide benchmarks for the AFF population from which subsequent analyses and comparisons can be made in the future. Key findings of this annual report are summarized below, under the research questions that were examined in this report. What Has Been the Pattern of Referrals to the AFF Program? Referrals to the AFF program continue to remain high, with an average of 783 referrals per quarter during the past year, despite changes in referral priorities. What Are the Characteristics of Participating Clients? The profile of a participating AFF client is one in which the client was predominately female (69%), Caucasian (65%), and about 30 years of age. About one-fifth of clients were Latino. Most clients had at least a high school diploma or GED, and over half were either employed, in school, or participating in a training program. About eight out of ten AFF clients were provided services funded through the Arizona Health Care Cost Containment System and delivered by their local RBHA. What Do We Know About Drug Use Among AFF Clients? It should come as no surprise that substance use is the overriding issue that brings these clients into the AFF program. The major substances that clients deal with, in order of reported use, were methamphetamines (40%), alcohol (32%), marijuana (26%), and cocaine (13%). Six out of ten clients (63%) were poly-substance users. The high use of methamphetamines is alarming, and should give pause to policy makers and substance use treatment providers to examine and ensure that the type of intervention delivered to this stimulant user population is appropriate and evidence-based. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 7 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 To What Extent Are AFF Clients Engaged in Substance Abuse Treatment? Engagement in treatment services was one of the suggested performance measures by the initial AFF program Steering Committee. About half of the referrals to the program resulted in an assessment and some type of further service. Among new clients for the reporting period, the median number of days from referral to assessment was slightly more than three weeks (25 days), and most began treatment services shortly thereafter. While most of the providers initiated outreach to a client within 24-hours of the referral, it was often the case that clients were difficult to locate, thus lengthening the engagement phase. To What Extent Are AFF Clients Staying in Treatment Services? Nearly half (48%) of participating clients remained in treatment for six months or longer, and 13% of clients remained in treatment less than 30 days. These patterns are promising, given that research on substance abuse treatment emphasizes that the longer a client remains in treatment, the more likely it is that treatment will result in long-term behavior change. What Are Some of the Child Welfare Outcomes Based on Available Data? The findings from this year’s evaluation would suggest that the recurrence of child abuse and/or neglect among CPS families participating in the AFF program is low. Only 4% of AFF clients had substantiated reports. The most frequent type of maltreatment substantiated was for neglect. Among children in foster care with parents or guardians participating in the AFF program, 16% of the children were reunified with their parents or guardians. Among children reunified with their parents or guardians, children who were removed from a parent after the parent’s referral to the AFF program spent significantly less time in foster care (median days = 85 days) compared to those children placed in foster care before or at the time of their parent’s referral to the AFF program (median days = 211). What Do We Know About Stakeholders’ Experiences with the AFF Program? Suggestions for improvement identified by AFF stakeholders included: increased communication between CPS and AFF staffs at the time of referrals; increased clarification among all stakeholders regarding the essential element of “family centered” education and treatment; increased case coordination and collaboration among all stakeholders; increased availability of residential treatment beds; and increased availability of transitional and affordable housing across the state. What Do We Know About Clients’ Satisfaction and Experiences with the AFF Program? In general, clients reported being pleased with services received through the AFF program, and perceived the services to be delivered in a timely, available and accessible Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 8 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 manner. Transportation continues to be a major barrier for some clients when accessing services, especially in rural areas. Conclusions and Recommendations Overall, the AFF program continues to meet and improve upon the outcome goals identified in ARS 8-884 of increasing the availability, timeliness and accessibility of substance abuse treatment to: 1) improve child safety, family stability and permanency for children in foster care or other out-of-home placement, with a preference for reunification with the child’s birth family; 2) achieve self-sufficiency through employment; and 3) promote recovery from alcohol and drug problems. Levels of engagement in treatment continue to be moderately high, with two-thirds of all referrals leading to assessments, and most clients receiving some treatment services after assessment. Findings on retention indicate that clients remain in treatment for several months, which is an expected proximal outcome on the road to recovery. This year’s outcome data have provided general benchmarks for the AFF population with respect to subsequent substantiated reports of abuse and neglect, reunification, and time spent in foster care. Based on the number of AFF clients using methamphetamines and other stimulants, the DES/AFF program staff may want to consider strategies that enhance the use of evidence-based treatment practices among AFF providers. There are effective treatment approaches targeting methamphetamine users that DES/AFF program staff may want to consider. In general, evidence-based treatment practices are identified as those treatment practices that have been demonstrated to be effective based on: 1) clinical trials research; 2) research appearing in peerreviewed professional journals; or 3) consensus-based guidelines developed by clinical, research, and administrative experts in the field. Sources for the identification of evidence-based treatment practices can be found in the National Registry of Effective Programs and Practices developed by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention, Evidence-Based Practice Toolkits developed by SAMHSA’s Center for Mental Health Services, and various publications such as CSAT Treatment Improvement Protocols, National Institute on Drug Abuse manuals, and National Institute on Alcohol Abuse and Alcoholism publications. The AFF program providers continue to improve upon client engagement rates, increasing the assessment-to-referral ratio, and decreasing the length of time from referral to assessment. We recognize that AFF providers face challenges when engaging CPS clients for treatment services, especially with those cases where the client may be resistant to engagement attempts. Enhanced engagement strategies might include the use of the Motivational Interviewing approach advocated by William Miller and Stephen Rollnick1 in which one of the clinician’s tasks is to influence a client’s state of readiness or eagerness to change. The use and effectiveness of motivational interviewing is well documented in the substance abuse literature and is an evidence-based practice referred to earlier. Further refinements to the AFF process would include additional guidelines to AFF providers that enhance the existing program exit protocol so that a consistent data set is collected upon 1 Miller, W.R. & Rollnick, S. (1991). Motivational Interviewing. New York, The Guilford Press. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 9 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 client program exit. In addition, there is current evidence that AFF providers are adhering more consistently to the evaluation data collection protocol established by the DES/AFF program staff, resulting in improved data quality and consistency. Finally, AFF providers should continue coordinating with the RBHA system to move eligible clients into the RBHA system in order to maximize resources and Title XIX funds. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 10 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 1. INTRODUCTION A rizona Families F.I.R.S.T. (Families in Recovery Succeeding Together) was established as a community substance abuse prevention and treatment program by ARS 8-881 (Senate Bill 1280, which passed in the 2000 legislative session). Under the requirements of the Joint Substance Abuse Treatment fund that was established under the legislation, Section 8-884 requires an annual evaluation of the Arizona Families F.I.R.S.T. program (AFF). The evaluation of AFF focuses on the implementation of community substance abuse prevention and treatment programs at each of the nine AFF site across the State of Arizona, the factors that contribute to their success, and the extent to which outcome goals identified in the enabling legislation have been attained. 1.1 Background Information on the Arizona Families F.I.R.S.T. Program The AFF program is administered jointly by the Arizona Department of Economic Security/Division of Children, Youth and Families (ADES/DCYF) and the Arizona Department of Health Services/Division of Behavioral Health Services (ADHS/DBHS), with DES designated as the lead agency. The legislation established a statewide program for substance-abusing families entering the child welfare system as well as those families receiving cash assistance through Temporary Assistance for Needy Families (TANF). The legislation recognized that substance abuse in families is a major problem contributing to child abuse and neglect, and that substance abuse can present significant barriers for those attempting to reenter the job market or maintain employment. Federal priorities under the Adoption and Safe Families Act (ASFA) that address child welfare outcomes, such as permanency and shorter time frames for reunification, coupled with lime limits established under the TANF block grant, also were factors behind the legislation. The purpose of AFF is to develop community partnerships and programs for families whose substance abuse is a barrier to maintaining, preserving, or reunifying the family, or is a barrier to maintaining self-sufficiency in the workplace. The joint Substance Abuse Treatment Fund was established to coordinate efforts in providing a continuum of services that are: family-centered; child focused; comprehensive; coordinated; flexible; community based; accessible; and, culturally responsive. These services were to be developed through government and community partnerships with service providers (including subcontractors and the RBHAs) and other agencies such as faith-based organizations, domestic violence agencies, and social service agencies. The Arizona Legislature mandated in ARS 8-884 that the following outcome goals be evaluated: • Increase the availability, timeliness, and accessibility of substance abuse treatment to improve child safety, family stability, and permanency for children in foster care or other out-of-home placement, with a preference for reunification with the child’s birth family. • Increase the availability, timeliness and accessibility of substance abuse treatment to achieve self-sufficiency through employment. • Increase the availability, timeliness and accessibility of substance abuse treatment to promote recovery from alcohol and drug problems. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 11 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 The initial AFF program Steering Committee2 required that the following performance measures be used to evaluate the effectiveness of the program: • Reduction in the recurrence of child abuse and/or neglect. • Increase in the number of families either obtaining or maintaining employment. • Decrease in the frequency of alcohol and/or drug use. • Decrease in the number of days in foster care per child. • Increase in the number of children in out-of-home care who achieve permanency. In the spring of 2001, nine provider agencies received contracts through DES to implement a community substance abuse prevention and treatment program under Arizona Families F.I.R.S.T. Contract providers across the State of Arizona were funded so that AFF services were available in every county. The AFF provider agencies and the geographic areas they service are summarized in the following table. Table 1.1.1 AFF Provider Agencies, DES District, and Counties Served AFF Provider Agency DES District County TERROS I Maricopa Community Partnership of Southern II Pima Arizona (CPSA) Arizona Partnership for Children III Coconino (AZPAC-Coconino) Arizona Partnership for Children III Yavapai (AZPAC-Yavapai) Old Concho Community Assistance III Apache and Navajo Center Arizona Partnership for Children IV Yuma (AZPAC-Yuma) WestCare Arizona IV La Paz and Mohave Horizon Human Services V Gila and Pinal Southern Arizona Behavioral Health VI Cochise, Graham, Services (SEABHS) Greenlee, and Santa Cruz Among the nine AFF providers, four are Title XIX providers (TERROS, CPSA, Horizon, and SEABHS) and provide treatment services for both Title XIX and non-Title XIX AFF clients. The remaining five providers are non-Title XIX providers (AZPAC-Coconino, AZPAC-Yavapai, AZPAC-Yuma, Old Concho, and WestCare) and must refer Title XIX AFF clients to the local RBHA or a Title XIX provider for treatment services. 2 The initial AFF program Steering Committee was a policy committee chaired by the Governor’s office that provided guidance and oversight to the program during the start-up phase of the program. The committee disbanded after the initial start-up year of program operations. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 12 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 1.2 Current Statewide Context of AFF Program Operations Throughout the first half of the reporting period, referrals to the AFF program were prioritized for CPS dependency cases and substance-endangered newborns (SEN). Beginning January 5, 2004, DES expanded eligible referrals to include CPS “service only” cases where there was an active CPS case plan requiring substance abuse treatment services. The DES/J.O.B.S. program also provided referrals to the AFF program. 1.3 Overview of the Evaluation Framework The evaluation design developed for the AFF program included both a process component and an outcome component. The process component focuses on program implementation to determine whether AFF provider agencies implemented the service model as intended by the legislation and program administrators. The process component is also useful for explaining why outcomes were or were not achieved. The outcome component was designed to address whether the AFF outcome goals and performance measures were achieved as well as other outcomes in the areas of recovery, family stability, safety, permanency, self-sufficiency, and systems change. The evaluation framework guiding this year’s evaluation report is contained in Appendix A, page 50. It describes the outcome goals as defined in ARS 8-884, required performance measures, DES Strategic Plan Key Indicators, appropriate research questions, key variables, data sources, methods for data collection, timeframes, and proposed analyses. 1.4 Data Sources for the Annual Report This year’s annual report draws upon data from multiple sources. Four core principles guided the use of data sources for the AFF program evaluation: • Collect the least amount of data necessary in order to satisfactorily meet the legislatively mandated evaluation requirements; • Avoid duplicative data collection efforts; • Use existing administrative data and formats whenever possible; and • Respect the differing management information systems capabilities among the nine providers. While AFF providers were urged both by the ADES/DCYF/AFF administrative staff and the current and previous evaluators to use a common reporting format, the data were not always consistent nor complete. These data issues were well documented in the 2002-03 evaluation report,3 and actions have been taken to correct them beginning with the 2003-2004 fiscal year. Actions taken this year include: contracting with Applied Behavioral Health Policy at the University of Arizona for program evaluation services, revisions to the AFF provider data collection protocol, and revisions to the AFF provider evaluation database. The primary information used for the analysis of AFF program services was service utilization data obtained directly from the nine AFF providers. These data were collected by the AFF providers and sent to the evaluation team in a variety of electronic formats, and imported into a 3 See Appendix A Lessons Learned through AFF Evaluation in Evaluation of Arizona Families F.I.R.S.T. Annual Evaluation Report, dated December 17, 2003. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 13 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 client-level database developed and maintained by the evaluator. Service utilization data are reported for the annual reporting period that covers July 1, 2003 through June 30, 2004. For some service activities, data also are presented from program inception (March 2001) through June 30, 2004. Another data set used for the analysis of the AFF program was enrollment and encounter data obtained from ADHS/DBHS for services utilized by Title XIX AFF clients. ADHS/DBHS service utilization data are reported for the annual reporting period that covers July 1, 2003 through June 30, 2004. It should be noted that ADHS/DBHS service utilization data is constantly updated and added to by the RBHAs and their providers, and there is a reporting lag, from service delivery to appearance in the ADHS/DBHS information system, of anywhere from 30 days to 90 days. The service utilization data for Title XIX AFF clients is relatively complete through June 30, 2004 since ADHS/DBHS provided an additional data set in early January 2005. Two additional data sets used for this evaluation include the ADES CHILDS information system, which provides child welfare information, and the ADES JAZ/AZTEC information system, providing employment services information. These data are reported for the annual reporting period that covers July 1, 2003 through June 30, 2004. The third major source of data used for the analysis of the AFF program is AFF stakeholders. These stakeholders include AFF program managers, RBHA liaisons, CPS managers and case workers, AFF provider collaborators, and clients of the program. A variety of data collection methodologies were used with these stakeholders, including individual interviews, focus groups, and document reviews. The purpose for using this third data source was to document and assess programmatic successes, changes in program implementation, updates on collaborative partnerships, perceived barriers and facilitators to program implementation, changes in contextual issues, and other events that may have positively influenced service delivery. Additional detail regarding the specific methodologies used for this evaluation has been included in the following chapters where the findings of our analyses are presented. 1.5 Organization and Contents of the Annual Evaluation Report This report is divided into a series of five chapters. The current chapter provides an overview of the evaluation design and methodology. Chapter Two summarizes the characteristics of clients referred to the program from July 1, 2003 through June 30, 2004 and the characteristics of clients that actually participated in services during this period. Included in the service activity reporting is information on levels of engagement, treatment services utilized, and lengths of stay in treatment. Data in Chapter Two, in general, are presented at the level of the individual AFF provider agency as well as cross-site (i.e., statewide totals). Chapter Three presents available outcomes based on child welfare data extracted from the CHILDS, JAS/AZTEC, ADHS, and provider data systems. These data enabled the evaluation team to assess subsequent CPS reports of abuse and neglect, reunification from foster care, and employment among participating clients as of June 30, 2004. Chapter Four presents findings with respect to program implementation. These findings are the results of a systematic qualitative analysis that addressed program directors’ perceptions over time as well as those of other AFF stakeholders. Chapter Four also Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 14 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 addresses findings on client satisfaction based on focus groups with clients across the AFF provider sites. Finally, Chapter Five provides a summary and conclusion of the major findings presented in the annual evaluation report. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 15 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 2. DESCRIPTION OF ARIZONA FAMILIES F.I.R.S.T. CLIENTS AND SERVICES RECEIVED T his chapter of the annual report presents descriptive information about individuals referred to the AFF program for the State Fiscal Year beginning July 1, 2003 and ending June 30, 2004, and cumulatively since the program inception. Data are presented also for AFF participating clients, those clients who received services during the reporting period and who may have been referred at any time since program inception in March 2001. Information highlighted in this chapter includes: • Referrals • Engagement in treatment • Assessments • Services received • Substance use • Demographic characteristics Findings are summarized using tables, charts, and summary bullet points. In the exhibits that follow, most tables include percentages, which are reported in the body of the tables to allow for comparisons across AFF provider agencies or time periods. 2.1 Referrals to the AFF Program4 During the State Fiscal Year 2004, a total of 3,135 individuals were referred to the AFF program, averaging 783 referrals per quarter. As one might expect, the AFF providers for Maricopa and Pima Counties received the largest number of referrals (56% and 18%, respectively) among the nine AFF providers. Table 2.1.1 AFF Program Referrals by Providers and Quarters July 1, 2003 - June 30, 2004 Quarter Jul-Sep03 DES District Oct-Dec03 FY04 Totals Jan-Mar04 Apr-Jun04 Count Col % Count Col % Count Col % Count Col % Count Col % 434 64.0% 432 59.1% 417 51.9% 478 51.8% 1761 56.2% II - CPSA 86 12.7% 127 17.4% 134 16.7% 206 22.3% 553 17.6% III - AZPAC Coconino 10 1.5% 8 1.1% 29 3.6% 20 2.2% 67 2.1% III - AZPAC Yavapai 33 4.9% 25 3.4% 55 6.8% 42 4.6% 155 4.9% III - Old Concho 17 2.5% 34 4.7% 37 4.6% 35 3.8% 123 3.9% IV - AZPAC Yuma 16 2.4% 11 1.5% 22 2.7% 26 2.8% 75 2.4% IV - WestCare 20 2.9% 45 6.2% 36 4.5% 28 3.0% 129 4.1% V - Horizon 27 4.0% 25 3.4% 37 4.6% 30 3.3% 119 3.8% VI - SEABHS 35 5.2% 24 3.3% 37 4.6% 57 6.2% 153 4.9% 678 100% 731 100% 804 100% 922 100% 3135 100% I - TERROS FY04 Totals 4 The number of AFF referrals is based on provider billing records to DES/DCYF (n = 3,172) where the referral record was unduplicated (29 duplicated referrals due to client resident moves) and the SSN was valid (8 referrals with incomplete or invalid SSN). Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 16 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Since the inception of the program in spring of 2001, more than 7,850 individuals have been referred to an AFF provider. As shown in the following table, there has been a steady increase in the number of referrals through March 2003, followed by a decline through the end of 2003, and increasing referrals, again, through the first half of 2004. Table 2.1.2 Cumulative AFF Program Referrals March 2001 - June 30, 2004 DES Districts I II AFF Providers Quarters TERROS CPSA Unknown < Mar 01 Mar-Jun 01 Jul-Sep 01 Oct-Dec 01 Jan-Mar 02 Apr-Jun 02 Jul-Sep 02 Oct-Dec 02 Jan-Mar 03 Apr-Jun 03 Jul-Sep 03 Oct-Dec 03 Jan-Mar 04 Apr-Jun 04 Total 18 31 145 119 150 161 152 184 359 469 497 434 432 417 478 4046 III III AZPAC Coconino AZPAC Yavapai 1 1 1 1 15 9 8 10 16 17 11 10 8 29 20 157 17 1 21 14 31 18 10 23 20 32 10 33 25 55 42 352 0 10 102 114 104 104 136 148 170 156 162 86 127 134 206 1759 III IV Old AZPAC Concho Yuma 14 7 22 16 27 19 15 35 33 38 39 17 34 37 35 388 12 0 4 4 3 7 12 15 17 21 3 16 11 22 26 173 IV West Care 0 1 17 6 4 10 27 21 44 31 21 20 45 36 28 311 V VI Horizon SEABHS 30 3 31 31 17 4 3 28 31 34 31 27 25 37 30 362 8 0 13 17 6 17 25 12 11 26 30 25 24 37 57 308 All Sites 100 54 356 322 357 349 388 476 701 824 804 678 731 804 922 7866 The vast majority of AFF program referrals (97%) since the inception of the program came from Child Protective Services (CPS). These data are consistent with information that has been reported previously in quarterly and annual evaluation reports. As shown in the following table, since the inception of the program, Horizon (7%), AZPAC Coconino (6%), and Old Concho (5%) had the highest percentage of referrals from the JOBS program. Table 2.1.3 Cumulative AFF Program Referral Sources March 2001 – June 30, 2004 DES Districts I II III AZPAC Source TERROS CPSA Coconino 97% 2% 1% 4046 96% 1% 3% 1759 87% 7% 6% 157 95% 5% 0% 352 CPS Family Builders JOBS III III IV AZPAC Old AZPAC West Yavapai Concho Yuma 94% 1% 5% 388 99% 0% 1% 173 Prepared by: Applied Behavioral Health Policy / The University of Arizona IV V VI Care Horizon SEABHS Sites 98% 0% 2% 311 92% 1% 7% 362 98% 1% 1% 308 97% 1% 2% 7866 All Page 17 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 2.2 Characteristics of Individuals Referred to the AFF Program During State Fiscal Year 2004, six out of ten referrals (60%) to the AFF Program were women, and the average age of referred persons was 29.9 years. Nearly one-fourth of the referrals (24%) were persons of Hispanic or Latino descent, 61% were Caucasians, 6% African Americans, and 4% American Indians. Most persons referred to the AFF program were single at the time of referral. These findings are summarized on the next page. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 18 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Table 2.2.1 AFF Program Referral Demographic Characteristics July 1, 2003 – June 30, 2004 DES Districts Providers I II III III III IV IV TERROS CPSA AZPAC Coconino AZPAC Yavapai Old Concho AZPAC Yuma West Care 1257 504 108 445 40 27 95 60 80 43 50 25 78 51 95 24 94 59 29.67 28.23 31.39 30.36 28.97 27.08 32.15 30.37 32.32 30.71 30.38 29.88 31.33 30.11 26.95 25.47 29.10 26.51 11331 435 155 79 8 0 162 95 17 5 30 244 31 15 1 12 8 0 83 14 0 0 0 58 83 16 0 10 14 0 28 23 0 0 24 0 89 5 0 0 0 35 40 32 1 4 25 17 55 36 1 0 7 54 1704 671 175 110 116 359 61.4% 24.2% 6.3% 4.0% 4.2% 1323 199 189 50 1761 147 68 61 277 553 25 13 16 13 67 47 27 41 40 155 45 24 17 37 123 27 2 11 35 75 37 10 21 61 129 51 11 21 36 119 40 14 021 78 153 1742 368 398 627 3135 69.4% 14.7% 15.7% Gender Female Male Average Age (yrs) Median Age (yrs) Race/Ethnicity White Hispanic/Latino Black/African Am American Indian Other Unknown Marital Status Unmarried Married Divorced/Sep/Wid Unknown Column Totals 1 V Horizon SEABHS Multiple responses for Terros Prepared by: Applied Behavioral Health Policy / The University of Arizona VI Page 19 All Sites Totals Percent 1897 1238 60.5% 39.5% 29.93 28.29 100% Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 2.3 Assessments During State Fiscal Year 2004, a total of 1,763 individuals referred to the AFF program received assessments for substance abuse treatment during the reporting period, either from an AFF provider or a local RBHA provider. Assessment data were compiled from two principle sources: AFF providers and ADHS/DBHS enrollment data. Of the 1,763 assessment records, 21% (378) were unique records supplied by AFF providers, 37% (644) were unique records from ADHS/DBHS, and the remaining 42% (741) of the individuals assessed have records from both an AFF assessment and an ADHS/DBHS enrollment. Of those individuals with both an AFF assessment and ADHS/DBHS enrollment, 177 individuals were enrolled with the local RBHA at some time prior to the AFF assessment; the remaining 564 individuals were enrolled with the local RBHA subsequent to the AFF assessment. Nearly 60% of the assessments were conducted for individuals within Maricopa County, and an additional 17% of the assessments for individuals within Pima County. An additional 28 individuals referred in June 2004 were later assessed in July 2004, and will be reflected in next year’s annual report. Fifty individuals referred to the AFF program during the current reporting period were already receiving services through the local RBHA, and were assessed prior to July 1, 2003. Table 2.3.1 AFF Program Assessments5 July 1, 2003 - June 30, 2004 Quarter Jul-Sep03 DES Districts I - TERROS II - CPSA III - AZPAC Coconino Total Oct-Dec03 Total Jan-Mar04 Apr-Jun04 Count Col % Count Col % Count Col % Count Col % Count Col % 225 65.4% 240 59.9% 245 54.1% 332 58.8% 1042 59.1% 32 9.3% 69 17.2% 83 18.3% 112 19.8% 296 16.8% 7 2.0% 6 1.5% 20 4.4% 9 1.6% 42 2.4% III - AZPAC Yavapai 11 3.2% 19 4.7% 23 5.1% 34 6.0% 87 4.9% III - Old Concho 14 4.1% 14 3.5% 22 4.9% 11 1.9% 61 3.5% IV - AZPAC Yuma 5 1.5% 4 1.0% 7 1.5% 9 1.6% 25 1.4% IV - WestCare 7 2.0% 4 1.0% 7 1.5% 10 1.8% 28 1.6% V - Horizon 20 5.8% 19 4.7% 26 5.7% 19 3.4% 84 4.8% VI - SEABHS 23 6.7% 26 6.5% 20 4.4% 29 5.1% 98 5.6% 344 100% 401 100% 453 100% 565 100% 1763 100% Since the inception of the program in the spring of 2001, more than 5,350 individuals have received assessments for substance abuse treatment, or about two-thirds of all individuals referred to the AFF program. As shown on the following page, the proportion of referred individuals who received assessments over the past 12 quarters shows considerable variation, especially after March, 2003. Several factors may account for this variability. First, there was a 5 Data reflect a combination of DES funded and Title XIX funded assessment services. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 20 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 decrease in the amount of funding for the AFF program, beginning July 2003 compared to previous years, that may have reduced service availability, including assessments. Second, there was about an eight-month lapse in evaluation services and technical assistance to AFF providers that may have contributed to an under-reporting of evaluation data by the providers. During this lapse in evaluation services, there was no evaluation data monitoring system in place to adequately assist DES/DCYF in detecting inconsistency or underreporting among some of the providers. Third, there was a change in the administration process for the AFF assessment. The AFF program adopted the ADHS/DBHS Behavioral Health Assessment core instrument for use by some of its providers (TERROS, CPSA, and SEABHS) beginning January, 1, 2004; the remaining AFF agencies began using the core instrument July 1, 2004. Consequently, there was a mix of assessment instruments and procedures in use during the reporting period that may have been a source of confusion in reporting assessment information to the evaluation team. Table 2.3.2 Cumulative AFF Program Assessments March 2001 – June 30, 2004 DES Districts I II III III III Providers AZPAC AZPAC Old Quarters TERROS CPSA Coconino Yavapai Concho IV AZPAC Yuma IV West Care 1 4 6 3 3 16 10 14 11 14 5 4 7 9 107 1.9% 3 18 5 9 10 21 19 30 20 12 7 4 7 10 175 3.2% < Mar 01 Mar-Jun 01 Jul-Sep 01 Oct-Dec 01 Jan-Mar 02 Apr-Jun 02 Jul-Sep 02 Oct-Dec 02 Jan-Mar 03 Apr-Jun 03 Jul-Sep 03 Oct-Dec 03 Jan-Mar 04 Apr-Jun 04 Total 22 16 2 2 7 154 99 3 24 22 120 99 1 15 24 122 75 8 25 14 123 77 11 20 11 154 72 10 15 13 144 88 7 18 24 265 89 14 24 24 382 89 8 26 25 2881 911 13 24 23 225 32 7 11 14 240 69 6 19 14 245 83 20 23 22 332 112 9 34 11 2816 1091 119 280 248 52.6% 20.4% 2.1% 5.0% 4.5% % Total Assess 1 Revised downward from previous quarterly report 2.4 V VI Horizon SEABHS 4 34 29 23 10 8 13 22 26 18 20 19 26 19 271 4.9% 3 17 13 7 15 20 15 13 28 19 23 26 20 29 248 4.5% All Sites Assessment 60 375 312 286 280 329 338 495 615 502 344 401 453 565 5355 100.0% Characteristics of AFF Participating Clients AFF clients were considered to be participating clients if they had an assessment and/or service plan developed during or prior to the annual reporting period and were participating in services during the annual reporting period (July 1, 2003 – June 30, 2004)6. These clients could have been 6 In previous evaluation reports, the definition for participating clients was the completion of a service plan and receipt of services during the reporting period. It appears that AFF providers may have underreported to the evaluation team the number and date of service plan completions; therefore, in order to provide as close an Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 21 Ratio n/a 97% 80% 80% 85% 71% 71% 75% 62% 51% 55% 56% 61% 68% Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 referred to the AFF program during the annual reporting period or anytime prior to the reporting period. During State Fiscal Year 2004, a total of 2,290 individuals met the definition of AFF participating clients. As shown in the table below, two-thirds of the participating clients were new this year, and the remaining 33% were clients enrolled from previous years who continued to receive services during the current reporting period. As expected, TERROS had the largest share of participating clients at 62%. There is variation among AFF providers in the proportion of new versus continuing participating clients; for example, WestCare and AZPAC-Yuma have the smallest proportion of new-to-total participating clients at 32% (27/84) and 34% (22/65) respectively, compared to Horizon, with the largest proportion of new-to-total participating clients at 90% (78/87). Table 2.4.1 AFF Participating Clients July 1, 2003 – June 30, 2004 DES Districts I II III AZPAC Source TERROS CPSA Coconino 1038 376 1414 61.7% 121 25 146 6.4% 40 26 66 2.9% 72 104 176 7.7% New Continuing Total % of Total III III IV IV V VI AZPAC Old AZPAC West Yavapai Concho Yuma Care Horizon SEABHS Sites 60 30 90 3.9% 22 43 65 2.8% 27 57 84 3.7% 78 9 87 3.8% 83 79 162 7.1% 1541 749 2290 100% All During State Fiscal Year 2004, seven out of ten (69%) AFF participating clients were women; the average age was 29.9 years. Among participating clients with recorded demographic characteristics, over one-fifth of the participants (23%) were for persons of Hispanic or Latino descent, 65% were Caucasians, 5% African Americans, and 4% American Indians. More than 71% of the participating clients had at least a high school diploma or GED, and 46% were either employed, in school, or participating in a work training program. The average reported family size was 2.7 persons. Eight out of ten AFF participating clients (79%) received services through the RBHA system at some time during the current reporting period. These findings are summarized on the next page. What are the implications for DES/DCYF and ADHS/DBHS from this description of the demographic profile of AFF participating clients? First, it should be evident that the AFF client population does not mirror the state-wide population profile.7 The AFF target population is predominately female, younger, less educated, with limited resources. Further, the ethnic/racial profile of AFF clients differs from that of the state-wide profile: fewer persons of Hispanic/Latino origin or Native Americans, and more African-Americans. This would suggest that the manner in which treatment services are provided to AFF clients should be culturally appropriate and gender sensitive. approximation to the previous definitions as possible, AFF participants who completed an assessment and received services after the assessment data were deemed to be AFF participating clients. 7 See Profile of Arizona General Demographic Characteristics. (2000). U.S. Census Bureau. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 22 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 DES Districts I II Providers TERROS CPSA Table 2.4.2 AFF Participating Clients Demographic Characteristics July 1, 2003 – June 30, 2004 III III III IV IV V AZPAC AZPAC Coconino Yavapai Gender Female 1000 86 40 117 Male 414 60 26 59 Age Average Age (yrs) 29.8 31.1 29.5 30.9 Median Age (yrs) 28.5 30.4 27.3 29.3 Race/Ethnicity White 787 59 37 147 Hispanic/Latino 257 41 17 14 Black/African Am 93 9 1 0 American Indian 51 4 8 0 Asian 19 0 1 3 Other 22 2 0 7 Unknown 185 31 2 5 Marital Status Never married 738 54 25 56 Married 226 29 11 41 Divorced/Sep/Wid 220 39 14 40 Unknown 230 24 16 39 Education Level No HS diploma/GED 114 27 18 40 HS Diploma/GED 599 39 24 38 Post HS/GED 4 10 2 9 Unknown 697 70 22 89 Total 1414 146 66 176 1 Percents are calculated based on “total minus unknowns” Prepared by: Applied Behavioral Health Policy / The University of Arizona VI Old Concho AZPAC Yuma West Care 63 27 43 22 53 31 70 17 113 49 1585 705 30.6 29.1 29.9 29.3 28.8 26.4 27.4 25.8 29.6 27.5 29.9 28.5 68 16 0 5 0 0 1 24 37 0 0 0 2 2 78 4 1 1 0 0 0 39 29 0 4 0 1 14 96 49 4 0 3 7 3 1335 464 108 73 26 41 243 65.2% 22.7% 5.3% 3.6% 1.3% 2.0% 41 20 17 12 24 1 12 28 25 10 16 33 47 10 20 10 43 15 23 81 1053 363 401 473 58.0% 20.0% 22.1% 22 14 1 53 90 12 11 2 40 65 18 10 2 54 84 48 16 3 20 87 35 34 10 83 162 334 785 43 1128 2290 28.7% 67.6% 3.7% Horizon SEABHS Page 23 All Sites Percent1 Totals 69.2% 30.8% Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Table 2.4.1 (continued) DES Districts I Providers TERROS AFF Participating Clients Demographic Characteristics (continued) July 1, 2003 – June 30, 2004 II III III III IV IV V CPSA AZPAC Coconino Employment Status Employed/Training/Educ 583 57 20 Not Employed 581 20 33 Unknown 250 69 13 Family Size Average Family Size 2.67 3.81 2.53 Median Family Size 2.0 4.0 2.0 1414 146 66 Total 1 Percents are calculated based on “total minus unknowns” Prepared by: Applied Behavioral Health Policy / The University of Arizona VI AZPAC Yavapai Old Concho AZPAC Yuma West Care Horizon SEABHS All Sites Totals 53 90 33 14 35 41 9 20 36 13 32 39 14 42 31 43 80 39 806 933 551 1.75 1.0 176 2.46 1.0 90 2.63 2.5 65 1.72 1.0 84 3.49 3.0 87 2.42 2.0 162 2.69 2.0 2290 Page 24 Percent1 46.3% 53.7% Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 2.5 Substance Use Among AFF Participating Clients At Time of Enrollment This section of the report presents information on the reported use (client self-report) of alcohol and various drugs, as well as the poly-drug co-morbidity patterns, at the time of AFF enrollment (information obtained at time of assessment) among AFF participating clients.8 Overall, substance use records were available for 91% of AFF participating clients; about six out of ten participating clients reported substance use at the time of AFF enrollment. Of this group (n = 2,090 AFF participating clients), significant variation was noted between those clients assessed by TERROS, wherein only 54% of the clients reported substance use, compared to 88% of those individuals assessed by other providers. The reason for this sharp difference is noted in the footnote below.9 Among AFF participating clients, 40% reported using methamphetamines and other stimulants within the 30 days prior to their enrollment into the AFF program. Other frequently mentioned substances used were alcohol (32%), marijuana (26%), and to a lesser extent, cocaine (13%). Other reported drug use is shown in the following table. Table 2.5.1 Types of Substances Used by AFF Participating Clients 30 Days Prior to Enrollment ( n = 2,290 Participating Clients) DES Districts I II III III III IV Providers AZPAC AZPAC Old AZPAC Substances TERROS CPSA Coconino Yavapai Concho Yuma Total Participating Clients AFF Clients Reporting Use Alcohol Cocaine Marijuana Heroin/Narcotics PCP/Hallucinogens Meth/stimulants Barbiturates/seds Inhalants Other drugs IV West Care V VI All Sites Horizon SEABHS Totals % 1414 146 66 176 90 65 84 87 162 2290 100.0% 47.6%9 19.9% 8.3% 10.8% 2.1% 0.6% 28.3% 0.5% 0.2% 5.7% 93.8% 65.1% 54.1% 67.1% 9.6% 18.5% 40.4% 4.8% 4.8% 32.9% 89.4% 65.2% 25.8% 63.6% 9.1% 3.0% 63.6% 4.5% 3.0% 3.0% 90.9% 64.8% 17.0% 57.4% 11.9% 11.4% 76.1% 7.4% 1.1% 1.1% 76.7% 57.8% 7.8% 35.6% 6.7% 2.2% 46.7% 5.6% 2.2% 2.2% 69.2% 27.7% 15.4% 35.4% 4.6% 0.0% 53.8% 0.0% 0.0% 4.6% 82.1% 26.2% 2.4% 34.5% 3.6% 1.2% 63.1% 3.6% 0.0% 17.9% 95.4% 21.8% 3.4% 36.8% 2.3% 0.0% 66.7% 0.0% 0.0% 40.2% 93.8% 54.9% 22.2% 59.9% 3.7% 1.9% 54.9% 3.7% 0.0% 13.6% 1447 733 302 607 91 64 912 44 16 210 63.2% 32.0% 13.2% 26.5% 4.0% 2.8% 39.8% 1.9% 0.7% 9.2% There was considerable variation in client substance use by AFF provider site. Methamphetamine and other stimulant drug use by clients of AZPAC-Coconino, AZPAC-Yavapai, Horizon, and WestCare providers was extremely high, with over 60% of clients reporting use of these 8 The substance use data presented in this section aggregates information provided by AFF provider records and ADHS/DBHS enrollment records. 9 During the reporting period, the substance use assessment code "none" was automatically generated when an electronic client file was first created; this "auto" procedure resulted in inconsistent reporting regarding client substance abuse use at the time of intake. Consequently, substance use at the time of intake is electronically underreported for TERROS clients. The electronic intake procedure has been changed and retraining intake staff is taking place on new system procedures for reporting client substance use at time of intake. This issue should have reduced impact for the next reporting cycle. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 25 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 drugs. Alcohol use by clients of CPSA, AZPAC-Coconino, and AZPAC-Yavapai was also high at 65% each. Chart 2.5.2 Poly-Drug Use Among AFF Participating Clients Poly Drug Use Among AFF Participating Clients As shown in the accompanying figure ( n = 1448 reporting drug use) on the following page, nearly four out of ten AFF participating clients (37%) reported using only a single substance, while 39% reported using two substances and 24% reported using three or more substances. As shown in Chart 2.5.3 below, among AFF clients who used alcohol, 52% reported also using methamphetamines and other stimulants, 44% also used marijuana, and 38% also used other drugs. Among those who used marijuana, 64% also used methamphetamines and other stimulants, 53% also used alcohol, and 42% used other drugs. Finally, among AFF clients who used stimulants, 43% and 41% also reported using marijuana and alcohol respectively. Chart 2.5.3 Poly-Drug Use Patterns Among AFF Participating Clients at Time of Enrollment 4 substances 7% 5+ substances 4% 3 substances 13% Single Drug Use 37% 2 substances 39% 70% 64% 60% 53% 52% 50% 44% 42% 43% 38% 40% 28% 30% 20% 41% 28% 20% 10% 7% 0% Alcohol Users (n = 733) Marijuana Users (n = 607) Meth Users (n = 912) Substances Used Alcohol Use Marijuana Use Prepared by: Applied Behavioral Health Policy / The University of Arizona Meth Use OD Use Page 26 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 2.6 Engagement In Treatment Among Participating Clients Engagement in treatment services was one of the performance measures suggested by the initial AFF program Steering Committee at the inception of the program. Information on levels of engagement is presented separately for two reporting periods. Data are presented for clients referred to the AFF program during the annual reporting period (July 1, 2003 – June 30, 2004). As shown in the table on the following page, of the 3,135 referrals to the AFF program during the State Fiscal Year 2004, over half (56%) received an assessment either from an AFF provider and/or a provider of the local RBHA. The average and median lengths of time from referral to assessment (either conducted by an AFF provider or a provider of the local RBHA) were 43.2 and 25 days, respectively. Similarly, the average and median lengths of time from assessment to first treatment service were 6.5 and 0 days, respectively. Table 2.6.1 Engagement Rates Among AFF Clients During State Fiscal Year 2004 DES Districts I II III AZPAC TERROS CPSA Coconino 1761 1042 553 296 67 42 40.8 25.0 62.4 37.0 9.5 0 24.0 0 Providers Services # of Referrals10 # of Assessments11 Days from referral to assessment12 Average/median Days from assessment to 1st treatment service Average/median 2.7 III III IV IV V VI AZPAC Old AZPAC West Yavapai Concho Yuma Care Horizon SEABHS Sites 155 87 123 61 75 25 129 28 119 84 153 98 3135 1763 31.9 22.0 50.5 28.0 45.7 18.5 53.1 33.0 43.0 19.0 22.5 10.0 30.2 14.0 43.2 25.0 33.5 20.0 19.4 10.0 9.8 1.0 20.1 5.0 9.7 7.0 19.4 8.0 6.2 0 6.5 0 All Substance Abuse Treatment Services Among Participating Clients As specified in the AFF program requirements, provider agencies under contract with DES were expected to develop a comprehensive continuum of treatment services to support clients in their recovery. Through collaboration, Title XIX clients served through the RBHA system receive services that may include treatment, transportation, case management, supports, crisis intervention and medications. The treatment modalities provided by agencies under contract with DES include the following services. Substance Abuse Education: These services are short-term in duration and are appropriate for clients who are unwilling to commit to more intensive services. Attendance at substance abuse awareness groups and individual counseling to consider the effect of substance abuse in one’s life would be included under substance abuse 10 DES provides a single payment for each referral that includes outreach and engagement activities. Reflects a combination of DES and Title XIX funded assessments. 12 Reflects days from DES paid outreach and engagement activities to assessment. 11 Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 27 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 education. While clients who are eligible for Title XIX services wait for their approval and enrollment in the Arizona Health Care Cost Containment System (ACHCCS), substance abuse education services are available to these clients. Outpatient Treatment Services: Outpatient treatment services are intended for clients who can benefit from therapy, are highly motivated, and have a strong support system. These clients need a minimum level of intervention and other supports. Service providers are required to provide a minimum of three hours per week of individual or group treatment (or a combination of both). Intensive Outpatient Treatment Services: Intensive outpatient services are intended for clients who can benefit from structured therapeutic interventions, are motivated, and have some social supports. This continuum of services is appropriate for clients who need a moderate amount of therapy and supports. At a minimum, service providers are expected to provide nine hours per week of therapy for a minimum of eight weeks. This therapeutic involvement can include individual, group, and family therapy; substance abuse awareness; and social skills training. Residential Treatment: Residential treatment services are intended for clients who need an intensive amount of therapeutic and other supports to gain sobriety. These services include 24-hour care and supervision. Similar to intensive outpatient treatment, residential treatment can include individual counseling, group therapy, family therapy, substance abuse awareness, and social skills training. Residential treatment may include children residing with parents while the parents are in treatment. Aftercare Services: Aftercare services are provided for clients at the end of their treatment plan through the AFF provider. It should be noted that aftercare service is not a recognized service category within the ADHS/DBHS system. At a minimum, the aftercare plan includes a relapse prevention program, identification and linkage with supports in the community that encourage sobriety, and available interventions to assist clients in the event that relapse occurs. Development of the aftercare plan is expected to begin while the client is in treatment. It should be noted that while aftercare is not a billable service under the ADHS/DBHS covered services guide, there is an expectation that RBHA service plans will address recovery management and relapse management. Title XIX treatment services covered by ADHS/DBHS through the Title XIX system include, but are not limited to, treatment, transportation, case management, supports, crisis intervention, and medications. During the reporting period, 71% of AFF participating clients received outpatient services, 10% residential services, 5% intensive outpatient services, 4% substance abuse education services, and 7% aftercare services. In addition to these services, about three-fourths of the AFF participating clients (74%) received case management services (service coordination), and slightly less than half (45%) received one or more supportive services. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 28 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 As shown in the following table, the mix of services varied widely among the nine AFF providers. For example, 15% of SEABHS clients received residential services, compared to 2% of CPSA’s clients; on the other hand, WestCare provided substance abuse education to 32% of their clients, while TERROS provided this service to less than 1% of their clientele. Table 2.7.1 Types of Services Used by AFF Participating Clients During State Fiscal Year 2004 ( n = 2,290 Participating Clients) DES Districts I Providers Services TERROS Residential IOP OP SA Education Support Aftercare Services Coord Children's Svcs Medical Other Participating Clients 2.8 II CPSA III III AZPAC AZPAC Coconino Yavapai III Old Concho IV AZPAC Yuma IV West Care V VI Horizon SEABHS All Sites 10.3% 2.3% 71.2% 0.4% 38.8% 3.6% 75.4% 0.9% 26.5% 17.8% 6.8% 4.8% 69.9% 2.1% 8.9% 8.2% 41.8% 0.0% 8.2% 7.5% 6.1% 19.7% 59.1% 7.6% 80.3% 19.7% 90.9% 0.0% 6.1% 40.9% 13.6% 20.5% 59.7% 2.3% 78.4% 30.7% 85.8% 0.6% 20.5% 31.8% 5.6% 2.2% 76.7% 21.1% 63.3% 12.2% 68.9% 1.1% 24.4% 5.6% 4.6% 0.0% 66.2% 20.0% 66.2% 3.1% 49.2% 0.0% 16.9% 4.6% 10.7% 16.7% 65.5% 33.3% 69.0% 4.8% 54.8% 1.2% 7.1% 29.8% 2.3% 3.4% 57.5% 1.1% 32.2% 4.6% 74.7% 1.1% 10.3% 9.2% 15.4% 4.9% 94.4% 1.9% 56.8% 1.2% 96.3% 0.0% 42.6% 74.7% 9.9% 5.1% 70.9% 3.6% 45.0% 6.7% 74.2% 0.7% 23.8% 22.2% 1414 146 66 176 90 65 84 87 162 2290 Most Intensive Substance Abuse Treatment Services Among Participating Clients In order to better understand the patterns of service utilization and variation in treatment services, different treatment level groups were identified based on a hierarchical continuum from most intensive treatment type to least intensive treatment . The groups correspond to AFF treatment modalities. The hierarchical continuum was applied to clients’ treatment services for the 12-month annual reporting period (July 1, 2003 – June 30, 2004). Clients participating in treatment services during this reporting period were counted in only one group that represented the most “intensive” treatment that they had received during the 12 month period. Of the 2,290 clients participating in AFF services between July 1, 2003 and June 30, 2004, there were 226 (10%) clients whose most intensive service modality was residential treatment, 112 (5%) clients whose most intensive service modality was intensive outpatient, and 1,298 (57%) who received outpatient as their most intensive service modality. There were 654 participating clients who did not fall into one of these three intensive service modality groups, but instead received some other type of service, such as substance abuse education and/or support services. The following table presents information for participating clients in the intensive treatment service modalities and the secondary treatment services they received. Among the additional treatment service modalities, social supports refer to the supportive services intended to help in achieving sobriety, such as transportation, child care, peer support, and housing assistance. Among those clients receiving residential services, 63% also received outpatient services, 43% Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 29 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 received service coordination, and 35% received social supports. Among those receiving intensive outpatient services, over half (57%) also received outpatient services, and 43% received social supports. Among those receiving outpatient services, 63% also received service coordination, and 43% received social supports. Table 2.8.1 AFF Participating Clients’ Most Intensive Treatment Service Modality and Secondary Services Received July 1, 2003 – June 30, 2004 Intensive SA None Residential Outpatient Outpatient Education Intensive Number in Svc Modality 226 112 1298 46 608 Percent that also received these Secondary Services: Intensive Outpatient 7.5% Outpatient 63.3% 57.1% SA Education 1.3% 5.4% Social Supports 35.0% 42.9% Aftercare 2.2% 8.9% Service Coordination 42.9% 44.6% Medication 28.8% 17.0% Other Services 25.7% 19.6% 2.9 37.8% 5.5% 62.6% 23.6% 20.2% 47.8% 8.7% 52.2% 15.2% 15.2% 51.6% 9.9% 89.1% 17.1% 22.7% Time Spent in Treatment Among Participating Clients The table on the following page presents findings with respect to lengths of stay in treatment services for AFF participating clients during the current reporting period who had an opportunity to spend at least six months in treatment (i.e., had an assessment conducted by December 31, 2003). For clients participating in services during the annual reporting period and with an opportunity to spend at least six months in treatment, nearly half (48%) remained in treatment for six months or longer. Four out of ten participating clients (40%) spent between 30 days to 180 days in treatment, and 13% of the clients spent less than 30 days in treatment. Table 2.9.1 Treatment Retention: Time Spent in Treatment for Clients with at Least Six Months Opportunity # of Days in Treatment Clients <31 Days 75 31-90 Days 103 91-180 Days 135 181-270 Days 179 271-365 Days 98 >365 Days 7 Total 597 Prepared by: Applied Behavioral Health Policy / The University of Arizona % of Clients 12.6% 17.2% 22.6% 30.0% 16.4% 1.2% 100% Avg Days 7.8 61.2 133.0 222.9 306.6 853.0 168.8 Page 30 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Research on the effectiveness of substance abuse treatment programs has documented that a client’s length of stay in treatment is an important predictor of successful outcomes, with the typical result being that the longer a client stays in treatment, the better the outcome (e.g., the more likely it is that treatment will result in long-term behavior change).13 The findings presented here indicated that overall, AFF participating clients are engaged in treatment and are remaining in treatment for several months. These are intermediary outcomes of treatment success. 13 Hubbard, R., Mardsen, M., Rachal, J., Harwood, H., Cavanaugh, E., & Ginsberg, H. (1989). Drug Abuse Treatment: A National Study of Treatment Effectiveness. Chapel Hill: The University of North Carolina Press. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 31 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 3. DESCRIPTION OF OUTCOMES DATA T his chapter presents available outcomes data in the area of child welfare and employment for cohorts of participants in the AFF program who received treatment services during the annual reporting period. 3.1 Methodology The research questions that address issues regarding recurrence of child abuse and/or neglect, reunification from foster care, time spent in foster care, and self-sufficiency through employment were explored through an analysis of data on AFF participating clients who received services during the annual reporting period (July 1, 2003 – June 30, 2004). For child welfare data, two cohorts were examined. These cohorts were defined as follows: Cohort 1: This cohort included participating clients referred to the AFF program and participating in treatment services during the annual reporting period. Cohort 2: This cohort included participating clients referred to the AFF program prior to July 1, 2003 who participated in treatment services during the annual reporting period; Child welfare data was extracted from the DES CHILDS system to cover any reporting that occurred up through June 30, 2004. For employment data, outcomes are reported on all clients participating during the annual reporting period for whom employment status information was available at the time they enrolled in AFF, and at any subsequent follow-up points up to June 30, 2004. The data on employment were based on all possible data systems that contained employment information on AFF clients, including the DES JAS systems, ADHS system, and provider-level data. This analysis includes more clients than just those identified in the AFF client-level database system as “JOBS referrals.” Since it is recognized that a program such as AFF services “dual system” clients who may be both TANF recipients and involved in the child welfare system, the evaluation plan was developed to include in the self-sufficiency analyses all of the AFF clients for whom employment data were available. Findings are reported under major evaluation questions developed to address the legislative outcome goals, outcomes related to the DES strategic plan, and questions posed by the DES Director based on last year’s annual evaluation report. 3.2 Child Welfare Outcomes Among AFF Clients: Is There a Recurrence of Child Abuse and/or Neglect Among CPS Families Participating in AFF? This evaluation question examines whether AFF-participating clients identified in the CHILDS data system experience a substantiated report of child abuse or neglect after their enrollment in the AFF program. The following table summarizes the overall findings with respect to substantiated CPS reports among AFF participants. There were a total of 2,180 CPS referred Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 32 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 clients participating in the AFF program during the annual reporting period. As of June 30, 2004, only 4% of CPS referred clients had a substantiated report filed since their enrollment in AFF. While not directly comparable, during the reporting period October 1, 2003 – March 31, 2004, 12% of child abuse, neglect and abandonment reports resulted in a substantiated finding.14 Of the 93 AFF participating clients with substantiated reports, 62% occurred within six months following the client’s enrollment in AFF, and 38% of the reports occurred after six months of enrollment. Table 3.2.1 Substantiated CPS Reports Among CPS Referred AFF Participating Clients July 1, 2003 – June 30, 2004 DES Districts I II III III III IV Providers AZPAC AZPAC Old AZPAC CPS Reports TERROS CPSA Coconino Yavapai Concho Yuma Substantiated 4.0% 3.4% 6.1% 4.0% 6.7% 1.5% Reports No Subsequent or Substantiated 96.0% 96.7% 93.9% 96.0% 93.3% 98.5% Reports Total AFF Clients 1414 146 66 176 90 65 IV V VI West All Care Horizon SEABHS Sites 2.4% 1.1% 6.2% 4.1% 97.6% 98.9% 84 87 93.8% 162 93.9% 2290 There was a small amount of variation in the number of substantiated reports among the nine provider sites, ranging from a low of 1% for AFF participating clients in Pinal/Gila Counties to a high of 7% for AFF clients in Navajo and Apache Counties. Among the 93 AFF clients with substantiated reports, 13 clients received residential treatment, three clients received intensive outpatient treatment, and 54 clients received outpatient treatment as their most intensive treatment service modality; the remaining 23 clients received a variety of support services. With respect to reported substance usage, two-thirds of clients with substantiated neglect reports (69%) reported substance use (21 clients reported single drug-use and 43 clients reported polydrug use), while the remaining 29 clients self-reported no substance use. The table on the following page presents information on the types of child abuse and neglect associated with the CPS reports for the 93 AFF clients. The data indicate that the vast majority of substantiated reports were for neglect (98%), and an additional 24% were for physical abuse. These findings are consistent with other studies which showed that substance-abusing caregivers tend to be linked with neglect referrals rather than with sexual or physical abuse referrals.15 14 Arizona Department of Economic Security. (2004). Child Welfare Reporting Requirements: Semi Annual Report For the Period October 1, 2003 Through March 31, 2004. 15 Sun, A., Shillington, A.M., Hohman, M., & Jones, L. (2001). Caregiver AOD Use, Case Substantiation, and AOD Treatment: Studies Based on Two Southwestern Counties. Child Welfare, 80(2), 151-177. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 33 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Table 3.2.2 Substantiated CPS Reports: Types of Child Abuse/Neglect Among AFF Participating Families (n = 93 Clients with Substantiated Reports) Type of Maltreatment Frequency Percent Neglect 91 97.8% Physical Abuse 22 23.7% Sexual Abuse 6 6.5% Emotional Abuse 2 2.2% 3.3 Child Welfare Outcomes Among AFF Clients: Are Children in Foster Care Whose Caregivers Enroll in AFF Reunified with Their Caregivers? Data on foster care children whose caregivers were participants in the AFF program during the annual reporting period are presented in the following table. There were a total of 1,208 AFF participating clients with children placed in foster care. Of this total, 871 AFF clients were new clients (enrolled on or after July 1, 2003) with 1,519 unduplicated children in foster care placement, and 337 individuals were AFF continuing clients (enrolled prior to July 1, 2003) with 544 unduplicated children, representing a total of 2,063 unduplicated children in foster care. Table 3.3.1 Reunification of Children Placed in Foster Care Cohort 1 New Clients ( n = 871) Placed in Care Before or After AFF Referral Number of Unduplicated Children Reunified Still in Care Discharged Cohort 2 Total All Continuing Clients Participating Clients ( n = 337) ( n = 1,208) Before After Before After Before After 1292 12.8% 84.4% 2.8% 227 8.8% 85.9% 5.3% 379 27.7% 60.9% 11.3% 165 24.2% 69.1% 0.6% 1671 16.2% 79.0% 4.8% 392 15.3% 78.8% 5.9% Total 2063 330 1630 103 Among new AFF clients, there were 1,292 children placed in foster care prior to or at the time of referral to the AFF program, and 227 children placed in foster care after their caregivers’ referral to AFF. Over a third of the children (36%) placed in foster care prior to or at the time of their caregiver’s referral to AFF were reunified, compared to 23% of children placed in foster care after their caregiver’s referral to AFF. Among children of new AFF clients, the rate of reunification was 12%. An additional 3% of children were discharged from foster care for other reasons (e.g. adoption, guardianships, living with relatives, emancipation). Among continuing AFF clients, there were 379 children who had already been placed in foster care prior to or at the time of their caregiver’s referral to AFF, and 165 children placed in foster care after their caregiver’s referral to AFF. Among children in this cohort, 27% were reunified with their caregivers, 63%% were still in foster care, and 10% were discharged by June 30, 2004. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 34 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 The table on the following page examines the number of days that reunified children spent in foster care for both groups of children. In general, children placed in foster care after their parent or caregiver’s referral to AFF were reunified more quickly (85 median days in care), compared with children placed in foster care prior to their parent or caregiver’s referral to AFF (211 median days in care). This finding was more pronounced for children whose parent or guardian was new to the program (i.e. began receiving services after July 1, 2003), compared with children whose parent or guardian was a continuing client (i.e. had received services prior to July 1, 2003). Since reunification occurs more quickly among new AFF participants, one motivational strategy would be for providers to encourage clients in “change talk” that stresses the importance and value of reunification with their children, and elicit statements from the client that “connects” the goal of reunification with the successful completion of their treatment plan. Table 3.3.2 Days in Care For Children Reunified with Parent(s) or Caregiver Cohort 1 New Clients Placed in Care Before or After AFF Referral Children Reunified Minimum Days in Care Maximum Days in Care Median Days in Care Average Days in Care 3.4 Before 165 1 599 162.0 168.7 After 20 1 92 12.0 29.4 Cohort 2 Total All Continuing Clients Participating Clients Before 105 1 983 344.0 316.4 After 40 19 518 157.5 192.4 Before 270 1 983 211.0 226.1 After 60 1 518 85.0 138.0 Total 330 1 983 192.5 210.1 Employment Outcomes Among AFF Clients An underlying premise behind analyzing employment outcomes among AFF clients is that a substance abuse problem can interfere with work performance, and for some persons, can interfere with their ability to either maintain employment or obtain employment if currently unemployed. From our earlier demographic description of participating clients (see page 20-21), we know that 51% were employed and 49% unemployed at the time of enrollment into the program. In last year’s evaluation report, the analysis on employment outcomes was based on data available through the JAS/AZTEC systems, the ADHS system, and AFF provider data. Unfortunately for this year’s analysis of employment outcomes, a number of data issues recently became apparent that affect our ability to analyze and comment on client employment outcomes in a meaningful way. First, there appears to have been a change in the data collection protocol for the JAS/AZTEC system that occurred at the start of or before the current reporting period. There were no JAS/AZTEC records for AFF clients with: 1) JAS service dates after June 13, 2003; 2) JOBS status date after January 20, 2004 (only 13 records with dates between 7/1/03 and 1/20/04); 3) no hire dates later than June 9, 2003; and 4) no employment termination dates later than June 12, 2003. Secondly, while employment status is part of ADHS’s demographic data set required at intake, again at the annual review, and upon a significant change, actual comparisons would only exist for a subset of clients who received at least an annual review. Finally, there is no consistent data collection protocol in place among the AFF providers that would allow for a Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 35 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 meaningful comparison at a given point in time. This issue will be addressed and rectified for the next evaluation reporting period which began July 1, 2004. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 36 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 4. IMPLEMENTATION OF THE ARIZONA FAMILIES F.I.R.S.T. PROGRAM T he evaluation of the AFF program included the collection of process data through interviews and focus groups with AFF program directors, AFF collaborating program partners, RBHA representatives, CPS district mangers and case workers, and AFF clients. In this chapter, findings are presented from data collected between July – September, 2004. As such, this information addresses program implementation successes and challenges through June 30, 2004. 4.1 Methodology Fifteen personal or telephone interviews were conducted with AFF program staff from each of the nine AFF provider agencies using a semi-structured interview. The protocol was designed to systematically assess AFF program directors’ perceptions regarding changes in program implementation and contextual issues and events affecting the implementation of the AFF program. Additionally, AFF program staff were asked to discuss their perceptions of facilitators and barriers to client success and the status of collaborative partnerships over the past year. Focus groups were conducted with 52 persons representing AFF collaborating partner agencies using a semi-structured protocol designed to systematically assess perceptions of the AFF program and agency collaboration within each provider site. Client focus groups were conducted with 67 current and 10 former AFF clients from each of the nine provider sites, using a semi-structured protocol. The protocol was designed to systematically assess client satisfaction and experiences with AFF services. Each AFF provider agency assisted the evaluation team in arranging the meetings with clients. The criteria for inclusion in the current client focus groups was that clients needed to be enrolled in the AFF program and receiving some type of treatment services. Each client who participated in the focus groups was provided a $20.00 gift certificate to Target or Wal-Mart as an incentive for their participation. Questions posed to clients focused on their experience with AFF, their level of satisfaction with the program, the services they found most helpful, and areas in which they experienced continuing needs. In-person and telephone interviews were conducted with seven RBHA liaisons from late July through October 2004. The purpose for these interviews was to obtain RBHA representatives’ perceptions regarding facilitators and barriers to implementing the AFF program, changes in the way the RBHA has been involved in delivering services to AFF clients, and the status of their collaborative partnerships with their respective AFF providers. Finally, in-person or telephone interviews were conducted with five DES district CPS managers, and focus groups were conducted with 61 CPS staff and case workers in the six DES districts. The purpose of these interviews and focus groups was to obtain CPS representatives’ perceptions regarding the facilitators and barriers to implementing the AFF program, and regarding changes in the way CPS has been involved in the program, particularly through referrals and case coordination activities. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 37 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 A complete listing of qualitative study participants and study protocols is contained in Appendix B. 4.2 Timeliness of Service Delivery The vast majority of stakeholders acknowledged the ease and speed with which clients can be referred into and engaged with the AFF program. Within each of the six DES districts, there are designated AFF providers that serve as the point of contact for referrals and client engagement. As part of the outreach and engagement process, the designated AFF provider also initiates an inquiry with the Arizona Health Care Cost Containment System (AHCCCS) or the local RBHA to determine whether an individual is currently receiving services through either of these two systems. Once an individual is engaged and consents to services, an assessment, treatment plan, and treatment services are scheduled. For those referred individuals already receiving or eligible to receive services from AHCCCS or the local RBHA, a referral for services is immediately made to the local RBHA. Number of Days Between Referral and Screening. All AFF coordinators reported that they were aware of the requirement for outreach to occur within 24 hours of referral. Most CPS staff and some collaborating partner agencies were also aware of this requirement. For eight of the providers, AFF staff reported that they initiated outreach within the 24 hour requirement. All providers reported that they experienced delays in initiating outreach services if the referral information was incorrect or inadequate. One provider does not begin the referral process until all referral information is correct. Several AFF coordinators noted that CPS staff will assist in the outreach effort if requested by the AFF provider. Clients reported that they were usually aware that the referral had been made; they also reported that they may have contributed to delays in outreach by avoiding outreach workers. Eight of the nine AFF providers have a single point for the receipt of referrals within the region. In one region, some CPS staff referred clients directly to a non-AFF provider, which caused some confusion for enrolling clients according to that AFF provider’s referral process. CPS staff countered that the direct referral to a provider moves the client into intake and treatment at a more rapid rate. Number of Days Between Screening and Assessment. All AFF coordinators indicated that they were aware of the requirement for the assessment to occur within five working days of actual contact with the client and the client’s consent to services. All providers indicated that in most cases, once the client was located, they rapidly conducted a brief screening and scheduled an assessment, thus meeting the requirement. There were a range of approaches across AFF programs for conducting the outreach and screening. Number of Days Between Assessment and Service Delivery Plan. All AFF coordinators reported that the service delivery planning occurred within five days of completing the assessment. Three AFF providers reported that service delivery planning occurred at the time of the assessment, while the remaining six AFF Coordinators reported that service planning occurred at the time of intake to the treatment provider. In most areas, substance abuse service Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 38 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 delivery planning does not include participation of the client’s CPS case manager. Some clients reported being unaware of having completed a treatment plan, while other clients reported that they were told what their treatment would be, with little or no input from the client. Average Waiting Time for Appointments. AFF staff and clients reported that there was as little as 45 minutes to as long as a 30-day wait between referral and initial treatment for eight of the nine providers, depending upon the service modality. Most clients reported that they experienced no delay in getting appointments for treatment. If there was a request for inpatient or residential treatment, the wait time was often 30 days and longer. AFF staff and CPS case managers reported working with these clients in an effort to retain their motivation for treatment. Several treatment providers offered an outpatient group to these clients in an attempt to ready the client for residential treatment. Timing of Scheduled Transportation. All AFF coordinators, case managers and clients reported that many efforts were made to ensure that transportation was available for clients for intake and assessment appointments. CPS case managers also reported that if alerted by the client, they attempted to offer transportation assistance. It was noted that AFF outreach staff for several providers were willing go to a client’s home (even jail, if the client was incarcerated) in order to engage the client in treatment services. CPS Staff Perception of Time Frames for Receipt of Services. The AFF program has received broad praise in all regions for the timeliness and efficiency of the initial outreach and engagements steps of the AFF model. All reported that the ease using of a single referral form, as well as clarity regarding point-of-contact for the referral, has created an unusually accessible system. CPS staff in all regions expressed appreciation for the effectiveness of the model, and indicated their desire to receive information on the status of the clients after outreach and engagement have occurred. Client Perception of Time Frames for Receipt of Services. AFF clients in all regions did not seem surprised at the rapid pace of initial outreach and engagement. Some related that they were still engaged in substance use behaviors or usually very angry at CPS, and the speed and skill with they were engaged by AFF outreach workers allowed them to come to grips with their circumstances. Clients participating in all the focus groups noted that they appreciated the speed with which the outreach and screening occurred, as well as the prompt delivery of case management services from AFF providers. Barriers to Receiving Services. The major barriers reported for AFF outreach, assessment, and service planning include the following: • Inaccurate, incomplete or “stale” address, telephone or other identifying information used to locate the client. One provider noted that they will not initiate outreach efforts until all required information has been submitted from CPS, often delaying outreach efforts. Many clients interviewed acknowledged that they slowed down the location process because they were not yet ready to accept treatment services. • Extreme distances across a region, limited transportation, and limited AFF staffing created challenges for AFF staff travel to the client and client travel to the AFF provider for outreach engagement. At least one provider noted that it took several months of Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 39 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • • • • coordinating living arrangements for an AFF family in order to move their household from one county to another so that they could begin treatment. At least two AFF providers noted that intensive case management (lasting three months) was sometimes required in order to stabilize clients and proceed with intake to treatment. AFF staff from several providers noted that CPS often closes a non-dependency case immediately after intake into AFF. This is a concern, in that case closure often diminishes a client’s commitment to engaging in treatment. CPS staff from several regions noted a concern that if outreach proved difficult or a client failed to attend three treatment sessions, the client’s case was often closed by the AFF provider. In one agency, the AFF program had come to a near-halt due to a vacancy in the AFF program coordinator position. CPS staff in that region reported that they were currently referring clients straight to the RBHA or other substance abuse providers for treatment options. Role of Collaborative Partnerships. AFF and CPS staff in three regions noted that they team together, if needed, in order to engage the client at the time of outreach. In several regions, the AFF program staff contacted CPS in order to obtain more information for the outreach to proceed. In most regions of the state, CPS staff did not perceive a collaborative partnership with the AFF program providers. At a minimum, they requested timely follow-up information regarding the status of the AFF client. All CPS and AFF staff agreed that they would take the time necessary to meet together in the future in order to increase effective collaboration among themselves. 4.3 Availability of Drug Treatment Services Program Capacity. All AFF providers reported offering the following range of services: • Substance abuse education – usually one 90-minute session per week; • Out-patient group – usually one or two 90-minute sessions per week; • Intensive outpatient group – usually two or three three-hour sessions per week. In some regions, usually the urban areas, there were inpatient or residential treatment options available to AFF clients. These options did not exist in the most rural regions of the state, and were often difficult to access for all regions. Several AFF providers noted that they offer relapse prevention classes, as well as some form of aftercare upon completion of substance abuse treatment. Service Gaps. The following service gaps were noted by various respondents: • Representatives of four AFF service providers noted that there were few or no detoxification facilities available within their region; • AFF providers in all regions noted there was a lack of inpatient or residential facilities sufficient to meet client needs; • With the exception of two AFF providers, there appears to be confusion among AFF providers about the AFF requirement that treatment services are “family centered” and “engage the family in treatment and enhance the family members’ understanding of the treatment recovery process” with services provided to children that consist of “family Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 40 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 focused and child centered treatment.” This confusion comes about in attempts to provide “family centered” services where one program’s focus is the child (i.e., CPS and Children’s Services at ADHS/DBHS), and another program’s focus is the parent (i.e., AFF program). Respondents who voiced this concern all expected that this element (family centered services) was handled by CPS, since the substance-abusing parent was the “identified” AFF client, and not the “family.” Service Enhancements. One AFF provider noted that they were striving to increase the offering of intensive outpatient treatment within their region. In a different area, the AFF program coordinator has increased the number of transitional housing (“Sober Houses”) available to clients. Perception of Sufficiency of Community Services. All stakeholders (AFF providers, collaborators, and RBHA representatives) reported that there were many efforts underway to improve transportation services. These included bus passes, gas vouchers and taxi vouchers, as well as AFF case managers working with clients to secure their own transportation. All stakeholders agreed that transportation remained the major barrier for the AFF client in accessing services. Other community services provided to the AFF client included child care, clothing and food banks, as well as rent and utility assistance. Most stakeholders agreed that there is insufficient effective childcare available for AFF clients. Many clients noted relying on family and friends to help them with these childcare needs. In one AFF client focus group, the clients living in “Sober Houses” reported often helping one another. Clients’ Perception of Services Offered by the AFF Program. In all regions, clients indicated that they were pleased with their substance abuse education or outpatient groups. In some regions clients noted that it was their own responsibility “to get the most out of the treatment program and get their children back.” Some clients noted that there were other clients in the treatment setting actively using drugs or alcohol while in the treatment group. Clients’ Desire For Service Enhancements. In all regions clients indicated a desire for more individual counseling, as well as family or couples counseling. In a few regions, clients requested couples counseling, and either their CPS case manager or AFF case manager assisted them in obtaining this service. Clients’ Contact with Case Manager. For all AFF providers, most clients reported their AFF case managers being very helpful. Across the providers, there were clients who expressed frustration with the CPS staff. These clients did not perceive CPS staff as supportive, but rather as barriers to reuniting with their children. Several clients also expressed frustration with the “changing rules and requirements” for family reunification. However, these very same clients also recognized that CPS staff seemed overloaded with cases. In addition, AFF clients recognized that high staff turnover at CPS and AFF providers resulted in difficulties with continuity of client care. 4.4 Accessibility of Drug Treatment Services Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 41 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Available Treatment Slots. In all areas there was little difficulty in AFF clients receiving any form of outpatient treatment. Differences did arise regarding the “sorting” of Title XIX and non-Title XIX clients. In one area there was great concern that the Title XIX provider was oversubscribed, and entry into treatment was delayed three weeks or more. Another barrier expressed by AFF providers was the constant “churn” of clients on-and-off of Title XIX rolls, resulting in difficulties for ensuring client continuity of care. Of greatest concern to clients and CPS staff was the insufficient quantity of inpatient and residential treatment slots, as well as resistance from AFF treatment providers in assisting clients in accessing the few beds that are available. All stakeholders reported a concern with the lack of transitional and affordable housing. Transitional housing is needed for clients entering treatment but not in need of residential care, as well as clients finding themselves with a housing crisis. Both AFF and CPS staff noted a concern for the high numbers of methamphetamine clients. These clients were having more difficulty in achieving a clean and sober life, and also presented with additional complications due to their drug use patterns. Service Utilization. The majority of service use by AFF clients consisted of outpatient treatment. Most clients reported attending drug education or outpatient groups once weekly. For two of the providers, there appeared to be a higher utilization of intensive outpatient groups. All providers reported some use of residential treatment. Additional services provided included anger management classes, domestic violence groups, and relapse prevention. Clients also reported receiving individual counseling if they were in an intensive outpatient program. Some clients reported that parent education or training services were delivered through the CPS system. Wait Time. For eight of the providers, clients reported immediate access to outpatient treatment; most clients did not perceive a difference in service delivery based on their Title XIX status. The wait time for residential treatment could be as little as two weeks, or as long as six weeks. Hours of Operation. In the case of most AFF providers, clients reported options for day or evening groups. For several providers, clients noted that they had scheduling conflicts with their employer, and usually the employer accommodated their treatment schedule. Clients in all areas perceived that CPS, AFF and Probation/ Parole were unaware of the demands placed on the client to schedule counseling, child visitation, urinanalysis, work, and other required meetings. Clients expressed a desire for AFF stakeholders to “get on the same page with each other.” Perception of Clients’ Access to Services. Clients did not perceive any real barriers to outpatient treatments services. In the case of two providers, clients reported having genderspecific groups. For three providers, clients noted that Spanish language treatment groups or interpreters were available to the Spanish-speaking clients. At one agency, clients were concerned that there were not sufficient Spanish language options for them. Some clients expressed concern about the difficulty in accessing residential treatment services, and said that their CPS worker had wanted them placed in residential treatment. We are unable to determine at this time if these are isolated cases, or the circumstances of the treatment plan. Barriers to Receipt of Services. As described earlier, there appear to be barriers for some individuals in accessing inpatient and residential treatment services. We assume that providers Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 42 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 are using some type of “level of care criteria” when developing the treatment plan with the client. Perhaps there needs to be more explanation and discussion with the client on the reasons and rationale for a particular course of treatment. Also, there may be two aspects responsible for this “perceived” barrier. First, the AFF treatment staff reported being dedicated to a “least restrictive environment” treatment policy, which usually means the least intensive treatment strategy that is clinically appropriate for the individual. On the other hand, CPS staff reported that clients may often need the intensive dosage that occurs in a residential treatment setting, which usually means starting with the most intensive treatment strategy. While both treatment providers and CPS staff need to work cooperatively on behalf of their mutual client, both should respect and acknowledge their separate areas of expertise and responsibilities. Other reported service barriers include: • Lack of transportation or faulty transportation (taxi service late for client pick-up, client is reprimanded or even denied access to the treatment group due to taxi service’s tardiness). • Conflict with the work schedule; client cannot miss work or forfeit the pay for that day. • Lack of childcare support for clients with children in the home. • Unspecified “bureaucratic barriers” when providing services to Native Americans. 4.5 Organizational Collaboration and Case Coordination Role of Collaborative Partnerships. In areas where there is a Family Drug Court or a functioning Family Adult Child Team, there were good working relationships among CPS, AFF providers, RBHA providers, and other stakeholders that facilitated case coordination. Case coordination occurred frequently, and often included the client, client’s family, probation/parole staff, CPS, AFF staff and RBHA provider staff, as well as others representing the justice system. This did add some complexity to the process, particularly in situations where judges often attempted to direct the service planning efforts, causing stress between AFF and CPS staff. Overall, however, the benefits of this approach appear to outweigh any costs. In areas without this structured approach, there was little structured and intentional case coordination. While AFF staff provide CPS staff with monthly documentation of clients’ progress, CPS staff noted that the documentation contained little useful information. Both partners noted that they have had little communication in the past, and all would like to participate in improving this partnership. Interestingly, seven of the nine providers do have some form of regular “collaboration meeting.” These meetings usually occurred quarterly, and were meant to discuss and problem-solve system barriers. The participants in these meetings were the AFF program coordinator, AFF treatment providers, and CPS staff. Rarely were AFF clients present at these meetings. Representatives from these partner agencies did not seem to effectively disseminate information and outcomes from these meeting to their agency staff. Thus, the staff from most of the collaborating partner agencies do not fully understand either the requirements of the AFF program model or the role of their partner agency with AFF. When asked why developing a shared mental model was difficult, all respondents acknowledged that “staff turnover” was a barrier to their collaboration. For the “stable staff,” work loads were extreme, and they noted, “a meeting better be effective and worth their time, or they will not attend!” Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 43 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Role of Referral System. In all regions, stakeholders reported an improvement in the AFF referral process. Overall, when AFF had staff in place, the system became clear and efficient, so that CPS staff knew the forms and process for referrals. Contact with Case Manager. Most clients reported that the assistance of the AFF case manager was invaluable. This person often acted as the facilitator and advocate for entry into treatment, as well as the provider of necessary resources for clients “to get back on their feet.” This assistance is most apparent in rural regions. Clients, as well CPS staff, noted that case management support was less apparent in the metropolitan areas and in one rural area. Within the collaborating partner system there was a general lack of understanding about the roles of the CPS case manager and the AFF case manager. Improved Family Stability. CPS staff across Arizona held a great deal of hope for the AFF program; however, CPS staff in several regions noted that they had low family reunification rates with AFF clients. There was a lack of clarity in the AFF provider system as to the role of treatment providers regarding increased family stability and reunification. In general, the providers “guessed” that this was the role of CPS. In two areas, AFF program coordinators noted that this was a primary outcome for the AFF program, but one which would require “a change in mindset” in the provider community. CPS staff also reported that improved family stability and reunification were primary outcomes of the program, and they took responsibility for their role in trying to achieve these outcomes. Several AFF program coordinators, as well as CPS staff, reported desiring an improved integration of the child and family team with the adult team in the RBHA system. Clients’ Perceptions of Family Stability. AFF clients across the state reported experiencing an improvement in their family functioning and stability. Some AFF clients either reported having begun family reunification steps, or anticipated that they would begin this process within a year. As one client noted, “To succeed, all you have to do is work with CPS, and they will do all they can to assist you.” Other clients were not as sure about the helpfulness of CPS staff, reporting that CPS was “always changing the rules” regarding reunification with their children. Clients’ Perceptions of Ability to Achieve Self-Sufficiency. Most clients noted that they were aware of the self-sufficiency outcome as necessary in achieving family reunification, and for the most part desired to increase their self-sufficiency. However, there was little connection between substance abuse treatment and education and/or job training in achieving self-sufficiency. Neither AFF nor CPS staff were clear about roles and responsibilities in helping clients achieve this outcome. 4.6 Opportunities For Enhancement Based on the findings from the stakeholder interviews and client focus groups, a number of suggestions are provided for improving the implementation of the AFF program. Timeliness. Building on the strengths of the current performance across Arizona, both AFF and CPS staff recommended increased communication among themselves at the time of referral. AFF Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 44 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 staff requested that CPS send as much information on the client as possible, including complete client information on the referral form, as well as other collateral data. CPS staff requested that AFF staff provide information on the receipt of referral, transition to outreach, and results of client engagement. All stakeholders interviewed realized that timeliness was impacted by the ability to locate clients rapidly. High client mobility, as well as lack of telephones and extreme distances, combined to frustrate rapid outreach and engagement. When CPS staff and AFF staff communicate and work together to outreach and engage resistant clients, there is an increased probability of engaging clients into treatment. Once the client is engaged, there is a rapid transition to treatment if the treatment is in an outpatient modality. However, there is a uniform concern for the lack of inpatient and residential treatment options in every region of the state. CPS staff and AFF staff recommend that they increase communication regarding the recommendation of inpatient/ residential services in order to gain a shared image of the appropriateness of this treatment option. All stakeholders agree that there is a serious gap in the arena of transitional housing known as “ sober houses” for AFF client. Several AFF coordinators also noted that there is a need for effective detoxification services in their region. Availability. Regarding availability of treatment within a “family centered context,” there was uniform concern that this was an area for exploration by the AFF partners. While some treatment providers and CPS staff reported that they offered avenues for family therapy and couples counseling, there was not a clear concept of roles and responsibilities regarding the delivery of treatment between the partners. The partners need to clarify these roles and responsibilities in a manner that is both flexible for regional differences, and consistent enough to deliver a “family centered” approach within substance abuse treatment. Of particular concern was the need to ensure that the treatment for the substance-abusing parent “links” with the treatment for the CPS identified child(ren). Several AFF coordinators noted that they hoped to increase integration across the adult teams and the child and family teams in their region. Accessibility. While both CPS and AFF staff utilized transportation support strategies, the major barrier for the AFF client continued to be transportation. Exploring a solution to this problem will require expanding the circle of involved partners to include both private and public transportation providers. Problem-solving options included: prompt pick-up by transportation providers, expanding bus service routes, and partnering with another provider such as a school district through transportation contracts. Treatment providers in rural areas may also want to examine location of services and determine additional options for facilities, including utilizing community buildings for service delivery. Providers may also want to explore avenues for increasing gender- and culture-specific services. Case Coordination. There is a need to enhance the structure of case coordination. While staff were encouraged to utilize informal routes for sharing information and case coordination, it was apparent that increased structure is necessary to ensure effective case coordination for the AFF client. Examples from the northern area of the state demonstrated that the use of family-adultchild teams (FACT) or Family Drug Court created a consistent, team-based approach to case coordination. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 45 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Agency Collaboration. As with case coordination, there was a need and a commitment from all stakeholders to improve the structure of agency collaboration. This needs to include meetings at a local level with all stakeholders present. Clearly, travel distance was a barrier for these partners that could be alleviated through the use of the state’s telemedicine resources; however, local and regional meetings should occur regularly. Each provider-region should determine the frequency of their meetings. Each meeting should have a clearly defined purpose and approach to agenda development, as well as facilitation of the meeting. All meetings should be documented so that there is a written history of the learning and accomplishments of the partners. Developing a structured training scope and sequence for use by AFF and CPS staff would enable these staff to train new hires, as well as provide regular AFF refresher courses to existing staff. External Barriers. All staff from both AFF and CPS noted the need to address the barriers of transitional and affordable housing throughout the state. One AFF program coordinator noted that HUD was issuing an RFP for affordable housing initiatives, and that DES/AFF may want to explore this opportunity for funding with other state agencies. This would require additional partners, perhaps from local municipalities or the County governments. Some staff also noted that the structure of the funding system at the state level could benefit from a reexamination to enable increased “flexibility” in the use of Title XIX funding.16 All AFF providers would benefit from an exploration of all the community assets available and underutilized in their region in order to enhance service delivery. Several AFF program coordinators and clients noted that gaining self-sufficiency is difficult when the local economy is depressed. They would like to see some form of community economic development occur in their area. Clients in rural areas noted that although they wanted to remain in their communities, they had difficulty escaping the substance-abusing culture while at the same time attempting to reshape their lives and “reputations” as people living clean and sober. Of note were comments heard in the Prescott community, where both treatment provider staff and clients reported how helpful it is when the culture of the town supports “sober living.” 16 It should be noted that ADHS/DBHS significantly expanded its array of covered behavioral health services in October 2001, and has a process in place for the identification and consideration of requested changes. See Covered Behavioral Health Services Guide at http://www.azdhs.gov/bhs/covserv.htm. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 46 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 5. SUMMARY AND CONCLUSIONS This chapter summarizes the major evaluation findings and conclusions for the AFF program in its third year of operation. 5.1 Major Findings • Referrals to the AFF program continue to remain high, with an average of 783 referrals per quarter during the past year, despite changes in referral priorities. • Over half of all referrals to the program this year resulted in an assessment, a rate somewhat lower than reported in previous years. While it is difficult to pinpoint the cause for the decline in assessments, it may be the result of several factors: changes in referral priorities, a lapse in evaluation oversight, and confusion over the change in assessment procedures. Regardless of the cause for the decline, it is important for the providers to document the reasons why a referral does not lead to an assessment. • The profile of a participating AFF client is one in which the client was predominately female (69%), Caucasian (65%), and about 30 years of age. About one-fifth of clients were Latino. Most clients had at least a high school diploma or GED, and over half were either employed, in school, or participating in a training program. About eight out of ten AFF clients received services through AHCCCS or their local RBHA. • It should come as no surprise that substance use is the overriding issue that brings these clients into the AFF program. The major substances that clients deal with, in order of reported use, are methamphetamines (40%), alcohol (32%), marijuana (26%), and cocaine (13%). Six out of ten clients (63%) were poly-substance users. The high use of methamphetamines is alarming, and should give pause to policy makers and substance use treatment providers to examine and ensure that the type of intervention delivered to this stimulant user population is appropriate and evidencebased. • Engagement in treatment services was one of the suggested performance measures by the initial AFF program Steering Committee. About half of the referrals to the program resulted in an assessment and some type of further service. Among new clients for the reporting period, the median number of days from referral to assessment was slightly more than three weeks (25 days), and most began treatment services shortly thereafter. While most of the providers initiated outreach to a client within 24-hours of the referral, it was often the case that clients were difficult to locate, thus lengthening the engagement phase. • The predominant type of service used by participating clients was service coordination (74%), followed by outpatient treatment (71%), and support services (45%). Most participating clients (57%) received outpatient treatment as their most intensive treatment service, followed by residential treatment (10%) and intensive Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 47 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 outpatient services (5%). The remainder of clients were provided with a mix of nonintensive services. 5.2 • Nearly half (48%) of participating clients remained in treatment for six months or longer, and 13% of clients remained in treatment less than 30 days. These patterns are promising, given that research on substance abuse treatment emphasizes that the longer a client remains in treatment, the more likely it is that treatment will result in long-term behavior change. • The findings from this year’s evaluation would suggest that the recurrence of child abuse and/or neglect among CPS families participating in the AFF program is low. Only 4% of AFF clients had substantiated reports. The most frequent type of maltreatment substantiated was for neglect. • Among children in foster care with parents or guardians participating in the AFF program, 16% of the children were reunified with their parents or guardians. Among children reunified with their parents or guardians, children who were removed from a parent after the parent’s referral to the AFF program spent significantly less time in foster care (median days = 85 days) compared to those children placed in foster care before or at the time of their parent’s referral to the AFF program (median days = 211). • In general, clients reported being pleased with services received through the AFF program, and perceived the services to be delivered in a timely, available and accessible manner. Transportation continues to be a major barrier for some clients when accessing services, especially in rural areas. • Suggestions for improvement identified by AFF stakeholders included: increased communication between CPS and AFF staffs at the time of referrals; increased clarification among all stakeholders regarding the essential element of “family centered” education and treatment; increased case coordination and collaboration among all stakeholders; increased availability of residential treatment beds; and increased availability of transitional and affordable housing across the state. Conclusion and Recommendations Overall, the AFF program continues to meet and improve upon the outcome goals identified in ARS 8-884 of increasing the availability, timeliness and accessibility of substance abuse treatment to: 1) improve child safety, family stability and permanency for children in foster care or other out-of-home placement, with a preference for reunification with the child’s birth family; 2) achieve self-sufficiency through employment; and 3) promote recovery from alcohol and drug problems. Levels of engagement in treatment continue to be moderately high, with two-thirds of all referrals leading to assessments, and most clients receiving some treatment services after assessment. Findings on retention indicate that clients remain in treatment for several months, which is an expected proximal outcome on the road to recovery. This year’s outcome data have Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 48 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 provided general benchmarks for the AFF population with respect to subsequent substantiated reports of abuse and neglect, reunification, and time spent in foster care. Based on the number of AFF clients using methamphetamines and other stimulants, the DES/AFF program staff may want to consider strategies that enhance the use of evidence-based treatment practices among AFF providers. There are especially effective treatment approaches targeting methamphetamine users available to treatment providers that DES/AFF program staff may want to consider. In general, evidence-based treatment practices are identified as those treatment practices that have been demonstrated to be effective based on: 1) clinical trials research; 2) research appearing in peer-reviewed professional journals; or 3) consensus-based guidelines developed by clinical, research, and administrative experts in the field. Sources for the identification of evidence-based treatment practices can be found in the National Registry of Effective Programs and Practices developed by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention, Evidence-Based Practice Toolkits developed by SAMHSA’s Center for Mental Health Services, and various publications such as CSAT Treatment Improvement Protocols, National Institute on Drug Abuse manuals, and National Institute on Alcohol Abuse and Alcoholism publications. The AFF program providers continue to improve upon client engagement rates, increasing the assessment-to-referral ratio, and decreasing the length of time from referral to assessment. We recognize that AFF providers face challenges when engaging CPS clients for treatment services, especially with those cases where the client may be resistant to engagement attempts. Possible engagement strategies might include the use of the Motivational Interviewing approach advocated by William Miller and Stephen Rollnick17 in which one of the clinician’s tasks is to influence a client’s state of readiness or eagerness to change. The use and effectiveness of motivational interviewing is well documented in the substance abuse literature and is an evidence-based practice referred to earlier. Further refinements to the AFF process would include additional guidelines to AFF providers that enhance the existing program exit protocol so that a consistent data set is collected upon client program exit. In addition, there is current evidence that AFF providers are adhering more consistently to the evaluation data collection protocol established by the DES/AFF program staff, resulting in improved data quality and consistency. Finally, AFF providers should continue coordinating with the RBHA system to move eligible clients into the RBHA system in order to maximize resources and Title XIX funds. 17 Miller, W.R. & Rollnick, S. (1991). Motivational Interviewing. New York, The Guilford Press. Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 49 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 APPENDICES Appendix 1 Revised Evaluation Plan Appendix 2 Listing of Qualitative Study Participants and Protocols Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 50 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Arizona Family F.I.R.S.T. Program Evaluation Plan for Fiscal Year July 1, 2003 through June 30, 2004 Outcome Goals – ARS 8-884 1. Increase the availability, timeliness and accessibility of substance abuse treatment to improve child safety, family stability and permanency for children in foster care or other out of home placement with a preference for reunification with the child’s birth family. 2. Increase the availability, timeliness and accessibility of substance abuse treatment to achieve self-sufficiency through employment. 3. Increase the availability, timeliness and accessibility of substance abuse treatment to promote recovery from alcohol and drug problems Research Questions Variable Data Sources Method of Data Timeframe Proposed Collection Analysis Did the AFF program • Number of days between referral & AFF provider Provider electronic Monthly Descriptive improve the service data data files statistics screening; timeliness of drug • Number of days between screening and treatment services in assessment; ADHS/DBHS CIS ADHS/DBHS each catchment area? Annually • Number of days between assessment & How? data for RBHA electronic data files service plan completion providers • Number of days between service plan and first treatment service • Engagement rate: # receiving at least one treatment service / # of referrals x 100% • Retention Rates: 30 Days: 2+ treatment services within first 30 days; 60 Days: 2+ treatment services each 30 day period 90 Days: 2+ treatment services each 30 day period 180 Days: To be defined Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 51 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • • • • • • • • • Did the AFF program improve the availability of drug treatment services in each catchment area? How? • • • • • • Did the AFF program improve the accessibility of drug treatment services in each catchment area? How? • • • • • • • • • • Number of days between referral & screening; Number of days between screening and assessment; Number of days between assessment & service delivery plan. Average wait time for appointments; Timing of scheduled transportation Staff perception of time frames in which clients receive services Barriers to receiving services Role of collaborative partnerships Clients’ perceptions of time frames within which they receive services Program capacity Service gaps Service additions or deletions Perception of sufficiency of community’s services Clients’ perceptions of services offered by the program Clients’ perception of whether service needs are met Client contact with case manager Available slots Service utilization Wait time Hours of operation Transportation Perception of clients’ access to services Barriers to receiving services Role of collaborative partnerships Role of referral system Prepared by: Applied Behavioral Health Policy / The University of Arizona AFF program managers Interviews Annually Qualitative analyses AFF participants Focus groups Annually Qualitative analyses AFF program managers Interviews Annually AFF participants Focus groups Annually Qualitative analyses AFF program managers Interviews Annually Qualitative analyses Key stakeholders Key stakeholders Key stakeholders Page 52 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • • How did improvements in timeliness, availability, and accessibility affect child safety? How did improvements affect family stability and permanency for children in foster care or other out of home placement? • • • • • • • • • • Clients’ perceptions of whether they actually receive services they need Clients’ perceptions of how well they understand how service delivery stem operations Proximity of services Contact with case managers Subsequent allegations of abuse & neglect Subsequent birth with prenatal drug exposure AFF participants Focus groups Annually Qualitative analyses DES CHILDS data set DES electronic data file Annually Descriptive statistics Adoption Family reunification Guardianship Long-term foster care Child(ren) remaining at home while caregiver receives treatment Client perceptions of family stability DES CHILDS data set DES electronic data file Annually Descriptive statistics AFF participants Focus groups Annually Qualitative analyses Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 53 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 How did improvements result in the reunification with birth families for children who had been place in out of home care? How did improvements affect TANF participants’ ability to achieve selfsufficiency through employment? • Family reunification DES CHILDS data set DES electronic data file Annually Qualitative analyses • • • • Receipt of TANF Secured employment Maintain employment status for 90 days Lose employment status and regain TANF benefits Client perceptions of ability to achieve self-sufficiency Drug and alcohol use past 30 days JAS DES electronic data file Annually Descriptive statistics AZTEC Annually AFF participants DES electronic data file Focus groups ADHS/DBHS core assessment AFF Provider service data Descriptive statistics Qualitative analyses Longitudinal analysis • How did improvements promote recovery from drug and alcohol problems? • ADHS/DBHS CIS data for RBHA providers • Drug screens Prepared by: Applied Behavioral Health Policy / The University of Arizona AFF client drug screens Date file submitted by providers Page 54 Annually At initial assessment Change in status Every 12 months At closure Monthly Descriptive statistics Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Performance Measures – Scope of Work, III-1: Required Performance Measures: 1. Reduction in the recurrence of child abuse and/or neglect; 2. Decrease in the frequency of alcohol and/or drug use 3. Decrease in the number of days in foster care per child 4. Increase in the number of children in out-of-home care who achieve permanency Research Questions Variable Data Sources Was there a reduction in the recurrence of child abuse and/or neglect? For those who had abuse/neglect allegations at program entry, what percent subsequently had children placed in foster care Was there an increase in the number of families either obtaining or maintaining employment? Was there a decrease in the frequency of alcohol and/or drug use? Was there a decrease in the number of days in foster care per child? Was there an increase in the number of children in out-of- • Reports of suspected child abuse/neglect DES CHILDS data set Method of Data Collection DES electronic data file • • Reports of suspected child abuse/neglect Foster care entry DES CHILDS data set DES electronic data file Annually Descriptive statistics DES JAS data set DES AZTEC data set DES electronic data file Annually Descriptive statistics ADHS/DBHS core assessment AFF participant drug screens Date file submitted by providers At initial assessment Change in status Every 12 months At closure Annually Descriptive statistics Annually Descriptive statistics • • Length of time receiving TANF Average monthly amount received from TANF • Secured employment • Maintained employment at 90 day followup • Drug and alcohol use past 30 days • Drug screens • Days in foster care DES CHILDS data set DES electronic data file • • Reunification Adoption DES CHILDS data set DES electronic data file Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 55 Timeframe Annually Proposed Analysis Descriptive statistics Descriptive statistics Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 home care who achieved permanency? What percentage of clients successfully completed their treatment service plans? • Service plan completion Prepared by: Applied Behavioral Health Policy / The University of Arizona AFF Provider service data ADHS/DBHS CIS data for RBHA providers AFF Provider service data Monthly ADHS/DBHS CIS data for RBHA providers Annually Page 56 Descriptive statistics Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Scope of Work, III-4: DES Strategic Plan Key Indicators Research Questions Variable Goal 1: To promote recovery from alcohol and drug abuse for AFF program participants • • • • Goal #2: To reduce the recurrence of child abuse and neglect of AFF program participants’ children. Goal #3: To establish permanency for the children of AFF program participants • • Data Sources Number of referrals for substance abuse treatment Participants who have engaged in at least one therapeutic service Participants who have engaged in AFF treatment for 3 months Participants who have engaged in AFF treatment for 6 months Individuals referred who have engaged in substance abuse treatment program and do not have a subsequent substantiated CPS report after 6 months of enrollment. AFF Provider service data # of children of referred individuals who participate in substance abuse treatment that achieve permanency through reunification, adoption or guardianship following at least 6-months parental participation in the substance abuse treatment program. Prepared by: Applied Behavioral Health Policy / The University of Arizona Method of Data Collection AFF Provider electronic data files Timeframe Monthly ADHS/DBHS CIS data for RBHA providers ADHS/DBHS electronic data files Annually AFF provider service data AFF Provider electronic data files Monthly DES/CPS data set DES/CPS electronic data files Annually AFF provider service data AFF Provider electronic data files Monthly DES/CPS data set DES/CPS electronic data files Annually Page 57 Proposed Analysis Descriptive statistics Descriptive statistics Descriptive statistics Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 AFF Program Coordinator/Provider Interview Referrals • Describe the referral process utilized in your region. • What is useful about the process? • What presents difficulties with the process? • What has been the volume of referrals this past year? Outreach • Describe the outreach process utilized in your region? • What is useful about the process? • What presents difficulties with the process? • What is the time between referral and outreach activity? • What is the process for linking outreach to screening and treatment? Screening • What is the process in screening clients? • What tools do you use in screening? • What are the options for language in the tools? • What is the process for determining levels of care as well as support services? Assessment • What other assessments occur outside the screening process? • How do you determine the needs of family members? • How do you ensure that clients enter treatment? • How do you determine the additional needs of the client? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 58 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Treatment services • How do you determine the necessary level of care for a client? • How do you ensure the clients’ cultural and gender needs are met? • What is the typical course of treatment? • What are the treatment delivery options in your area? • How are jobs skill training and education linked to treatment? Service Coordination • How are Title XIX clients care coordinated in your region? • Describe collaboration between your RHBA, CPS worker and you ( or the AFF provider). Services for children • Who tracks the service delivery of treatment to the children of your clients? • How does your client and the service to the clients’ children link? • What is the level of utilization of these services? • Who is responsible to for the development of the health of the family in treatment? Family education • Who tracks the service delivery of treatment/ support services to the family? • What does family education consist of in your region? • Who provides this service? • What is the level of utilization of these services? • Who ensures strengthening family functioning in the provider network? Support services • What additional support services are available in your region? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 59 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • What is the level of utilization of these services? Billing issues and transitions between funders • How do you determine client eligibility for AFF treatment? • How does this eligibility status change over time? • How does this affect treatment delivery? • How does this affect your agency? • Who assists you in solving problems as they arise? Case coordination • Describe the flow of case coordination in your region • What is effective about the process? • What could be improved in the process? Organizational collaboration • Describe the structure of organization collaboration in your region • How is the structure useful for ensuring high quality outcomes? • How do you discuss barriers and challenges to effective outcomes? • How do you problem solve these challenges? • What could be done to improve your collaboration? Outcomes • What have been the types of client outcomes that you are tracking? • What are some examples of successful client outcomes this year? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 60 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 CPS Program Manager Interview 1. What is the mission of the AFF program? 2. Describe your program goals or outcomes 3. What are the most significant challenges in your region? 4. What are the most significant opportunities in your region? 5. What should be the results of this program? 6. Is this program successful? Successful because: Not successful because: 7. Within this program what internal systems should be assessed for potential improvement? 8. Within this program what external systems should be assessed for potential improvement? 9. What innovation should the program consider? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 61 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Focus Group Questions Collaborating Agencies Referrals • Describe the referral process utilized in your region • What is useful about the process? • What presents difficulties with the process? • What has been the volume of referrals this past year? Outreach • Describe the outreach process utilized in your region? • What is useful about the process? • What presents difficulties with the process • What is the time between referral and outreach activity? • What is the process for linking outreach to screening and treatment? Screening • What is the process in screening clients? • What tools do you use in screening? Be specific, please provide copies. Are they standardized? Wormed? • What are the options for language in the tools? Are these specific –Spanish versions? Are there Spanish speaking staff available? • What is the process for determining levels of care as well as support services? Assessment • Who does the assessment? • What is the time span between screening and assessment? • What other assessments occur outside the screening process? • How do you determine the needs of family members? • How do you ensure that clients enter treatment? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 62 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • How do you determine the additional needs of the client? Treatment services • How do you determine the necessary level of care for a client? • How do you ensure the clients’ cultural and gender needs are met? • What is the typical course of treatment? • What services do the clients in your region need that they can’t get? • What are the treatment delivery options in your area? • How are jobs skill training and education linked to treatment? Service Coordination • How are Title XIX clients care coordinated in your region? Services for children • Who tracks the service delivery of treatment to the children of your clients? • How does your client and the service to the clients’ children link? • What is the level of utilization of these services? • What services do the children in your region need that they can’t get? • Who is responsible to for the development of the health of the family in treatment? Family education • Who tracks the service delivery of treatment/ support services to the family? • What does family education consist of in your region? • Who provides this service? • What is the level of utilization of these services? • Who ensures strengthening family functioning in the provider network? Support services • What additional support services are available in your region? • What additional support services are needed but not available in your region? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 63 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • What is the level of utilization of these services? Billing issues and transitions between funders • How do you determine client eligibility for AFF treatment? • How does this eligibility status change over time? • How does this affect treatment delivery? • How does this affect your agency? • Who assists you in solving problems as they arise? Case coordination • Describe the flow of case coordination in your region • What is effective about the process? • What could be improved in the process? Organizational collaboration • Describe the structure of organization collaboration in your region • How is the structure useful for ensuring high quality outcomes? • How do you discuss barriers and challenges to effective outcomes? • How do you problem solve these challenges? • What could be done to improve your collaboration? Overall • What do you find unique about AFF? • What are the greatest strengths in your region? • What are the greatest weaknesses in your region? • If you could change one of them, what would it be? Revised: 7/14/2004 5:33 PM Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 64 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Focus Group Questions Participating clients– Draft 3.1 Indicators Addressing Timeliness, Availability, and Accessibility Timeliness • How long did it take for the following: Screening Assessment Development of the treatment plan Participation in a treatment program Participation in other counseling or services • Once you were referred (once the problem was recognized) as having a problem with substance abuse, how long did you have to wait to start the program/counseling services? • What was helpful about the manner in which the provider worked with you at the first contact/ screening? • What was not helpful? • When you called to schedule a counseling appointment, how long did you have to wait for that appointment? • Are there any reasons why you were not able to make your appointment(s)? • Describe the kinds of case management services you received? • What has been helpful about case management • What has not been helpful about case management? • What form of transportation did you use to get to your appointments? • Was transportation a concern for you? • Did the health center provide transportation to the counseling appointments? Did you use their provided transportation? If no, why? • If yes, did it meet your needs? Availability Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 65 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • After the first referral what was your wish for timing in obtaining your appointment with the provider? • Did the provider make any changes in the days or times that were available for the type of appointment you needed? • What were the varieties of treatment options that you had hoped to receive? ( substance abuse treatment, parenting skills, job training, counseling, case management? ) • How available were these options? ( substance abuse treatment, parenting skills, job training, counseling, case management?) • Were there any factors that made you ineligible for treatment? If yes, what were they? Accessibility • When you learned about this program/when you were told about this program, did you think that it could help you to reduce or stop drinking or using drugs? If no, why not? • Was printed material that you received explained to you in a way that you could understand? • How often did you use this program/ the counseling services? One time each week, once every two weeks, once a month, a couple times a year? • How long did you have to wait for an appointment? • How far would you need to travel to get to an appointment? • Did the hours of operation cause scheduling problems for you? • Were you unable to attend sessions during the days and/or times the provider is open? • Was there any conflict with your work schedule? • What are some of the things that stopped you from making an appointment/keeping an appointment? • Was language an issue? • Do you think this program/these services can motivate you or assist you in improving your health? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 66 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • • • Did this program improve the ability of you to parent your children? Was childcare an issue? Did you notice any improvements in the program since you began? Staff • Do you feel comfortable meeting and talking to the counseling staff? • Do they make you feel at ease? If no, why? • Did staff treat you with respect even if you did not speak fluent English? • Do you feel that the counseling staff is capable of/able to understand your problems? • Do you feel that the counseling staff is capable of/able to identify with you/understand the cultural differences • Do you feel comfortable discussing your problems with someone of a different ethnicity or race? • Do you think that the information you share while participating in the program will remain confidential? If no, why? • Do you know other people in your community to go to the same provider, but for other/different services? • Are you afraid that someone you know will see you at the provider or learn why you are there? Financial • Was there any cost for the counseling sessions or program? • Did this affect your decision as to whether or not you would participate in the program? • What funding source paid for your program? • Were you required to co-pay for services? • How much were you required to pay? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 67 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • Did you not attend counseling sessions or program because you could you not afford to miss paid hours at work? Revised: 7/14/2004 5:43 PM Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 68 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 Focus Group Questions Other or Non Participating clients– Draft 3.1 Indicators Addressing Timeliness, Availability, and Accessibility Timeliness • How long did it take for the following: Screening Assessment Development of the treatment plan Participation in a treatment program Participation in other counseling or services • Once you were referred (once the problem was recognized) as having a problem with substance abuse, how long did you have to wait to start the program/counseling services? • What was helpful about the manner in which the provider worked with you at the first contact/ screening? • What was not helpful? • When you called to schedule a counseling appointment, how long did you have to wait for that appointment? • Are there any reasons why you were not able to make your appointment(s)? • Describe the kinds of case management services you received? • What has been helpful about case management • What has not been helpful about case management? • What form of transportation did you use to get to your appointments? • Was transportation a concern for you? • Did the health center provide transportation to the counseling appointments? Did you use their provided transportation? If no, why? • If yes, did it meet your needs? Availability Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 69 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • After the first referral what was your wish for timing in obtaining your appointment with the provider? • Did the provider make any changes in the days or times that were available for the type of appointment you needed? • What were the varieties of treatment options that you had hoped to receive? ( substance abuse treatment, parenting skills, job training, counseling, case management? ) • How available were these options? ( substance abuse treatment, parenting skills, job training, counseling, case management?) • Were there any factors that made you ineligible for treatment? If yes, what were they? Accessibility • When you learned about this program/when you were told about this program, did you think that it could help you to reduce or stop drinking or using drugs? If no, why not? • Was printed material that you received explained to you in a way that you could understand? • How often did you use this program/ the counseling services? One time each week, once every two weeks, once a month, a couple times a year? • How long did you have to wait for an appointment? • How far would you need to travel to get to an appointment? • Did the hours of operation cause scheduling problems for you? • Were you unable to attend sessions during the days and/or times the provider is open? • Was there any conflict with your work schedule? • What are some of the things that stopped you from making an appointment/keeping an appointment? • Was language an issue? • Do you think this program/these services can motivate you or assist you in improving your health? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 70 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • • • Did this program improve the ability of you to parent your children? Was childcare an issue? Did you notice any improvements in the program since you began? Staff • Do you feel comfortable meeting and talking to the counseling staff? • Do they make you feel at ease? If no, why? • Did staff treat you with respect even if you did not speak fluent English? • Do you feel that the counseling staff is capable of/able to understand your problems? • Do you feel that the counseling staff is capable of/able to identify with you/understand the cultural differences • Do you feel comfortable discussing your problems with someone of a different ethnicity or race? • Do you think that the information you share while participating in the program will remain confidential? If no, why? • Do you know other people in your community to go to the same provider, but for other/different services? • Are you afraid that someone you know will see you at the provider or learn why you are there? Financial • Was there any cost for the counseling sessions or program? • Did this affect your decision as to whether or not you would participate in the program? • What funding source paid for your program? • Were you required to co-pay for services? • How much were you required to pay? Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 71 Arizona Families F.I.R.S.T. Program Annual Evaluation Report for 2003 – 2004 • Did you not attend counseling sessions or program because you could you not afford to miss paid hours at work? Revised: 7/14/2004 5:43 PM Prepared by: Applied Behavioral Health Policy / The University of Arizona Page 72