2009 Annual Consumer Survey Report July 30, 2010 Prepared by: Dara Cirincione, ADHS/DBHS Office of Performance Improvement Carolyn Dempsey, ADHS/DBHS Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 TABLE OF CONTENTS EXECUTIVE SUMMARY……………………………………………………………………………… ............ 3 SURVEY RESPONSE RATES .…………………………………………………………… ........ 4 INTRODUCTION………………………………………………………………………………...4 METHODOLOGY AND SURVEY ADMINISTRATION……………………………………………………………………………...5 DATA ANALYSIS……………………….………………………………………………………… .. 5 SURVEY FINDINGS................................................................ ........................................................... 5 STATEWIDE IMPROVEMENT OPPORTUNITIES……………………………………………… ............................................... 12 CONCLUSION…………………………………………………………………………………..13 ATTACHMENTS Attachment A: 2009 Consumer Survey Protocol Attachment B: 2009 Adult Demographics Attachment C: 2009 Adult Line Item Attachment D: 2009 YSS-F Demographics Attachment E: 2009 YSS-F Line Item Attachment F: 2009 Raw Data Attachment G: 2009 Tribal Behavioral Health Survey Results APPENDICES YEAR 2009 ADULT CONSUMER SURVEY (ENGLISH AND SPANISH) YEAR 2009 YOUTH CONSUMER SURVEY FOR FAMILIES (ENGLISH AND SPANISH) 2009 Consumer Survey Report July 30, 2010 I. Executive Summary The statewide consumer survey was conducted during May, June and July 2009, jointly by the Arizona Department of Health Services, Division of Behavioral Health Services (ADHS/DBHS), Tribal/Regional Behavioral Health Authorities (T/RBHAs) and contracted service providers across the state. Two distinct surveys were administered based on the Substance Abuse and Mental Health Services Administration’s (SAMSHA) Mental Health Statistics Improvement Program (MHSIP) consumer surveys: The Adult Consumer Survey; and The Youth Services Survey for Families (YSS-F). The surveys solicit independent feedback from Title XIX/XXI adults and families of youth receiving services through Arizona’s publicly funded behavioral health system. The surveys measure consumers’ perceptions of behavioral health services in relation to the following domains: General Satisfaction Access to Services Service Quality/Appropriateness Participation in Treatment Outcomes Cultural Sensitivity Improved Functioning Social Connectedness Positive findings from the 2009 Adult Consumer Survey and YSS-F include: General Satisfaction yielded an 84% positive response rate from adult respondents; Participation in Treatment Planning was perceived positively by respondents in both adult and child programming at 82% and 89%, respectively; Service Quality and Appropriateness remained positive at 90% for child programming and received a positive response rate of 89% for adults; and Cultural Sensitivity continued to be positively perceived by children’s families at a rate of 90% Consumers are reporting increased satisfaction with most survey domains. Perception of the quality and appropriateness of the services received remains positive, with both adults and the families of child behavioral health recipients indicating they are involved in the planning of their treatment and by an increase in general satisfaction for both populations. The analyses of specific responses to each domain’s questions in the Adult and YSS-F surveys are discussed in detail within the body of the report. 2009 Consumer Survey data is compared to findings from previous years for comparative purposes in both the adult and child analyses. A review of survey data from 2008 to 2009 indicates that families of children receiving services reported higher positive response rates to six of the seven domains and no change in one of the domains, with increases from two to eight percentage points. Adult respondents to the 2009 Division of Quality Management Operations Office of Performance Improvement 3 2009 Consumer Survey Report July 30, 2010 survey also indicated an increase in positive responses for four of the seven domains with no change in two of the domains and only one showing a decreased rate. II. Survey Response Rates The MHSIP Consumer Surveys were offered to a statewide sample of 5,800 RBHA- enrolled consumers. The response rate is calculated by dividing the number of surveys returned by the number of surveys in the sample. In previous years, the rate was calculated by dividing the number of surveys returned by the number of surveys offered. This change in calculation resulted in a lower response rate for 2009. The statewide response rate was 47%, with rates ranging from 28% to 63% per RBHA. RBHA Sample (a) Adult & Youth Surveys Returned (b) Response Rate (b) / (a) Cenpatico-2 900 463 51% Cenpatico-4 1000 276 28% CPSA-3 900 442 49% CPSA-5 1000 535 54% Magellan 1000 374 37% NARBHA 1000 627 63% Statewide RBHA 5800 2717 47% III. Introduction The Arizona Department of Health Services, Division of Behavioral Health Services (ADHS/DBHS), the Regional Behavioral Health Authorities (RBHAs), in collaboration with their contracted providers, administered the statewide consumer survey May, June, and July 2009. As in the past survey cycles, the surveys are primarily based on the Adult Consumer Survey and Youth Services Survey for Families (YSS-F), recommended by the Mental Health Statistics Improvement Program (MHSIP). The use of the survey data to inform service delivery is promoted through dissemination of the survey results throughout ADHS/DBHS; review of survey data in the ADHS/DBHS Quality Management (QM) Committee meeting; incorporation of monitoring of survey indicators in RBHA contracts and QM Plans; and dissemination of survey results to ADHS/DBHS stakeholders and consumers via the ADHS/ DBHS website. Survey outcomes are reported to the National Association of State Mental Health Program Directors’ (NASMHPD) National Research Institute (NRI), Western States Decision Support Group (WSDSG), and to the 4 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 Substance Abuse and Mental Health Services Administration’s (SAMHSA) Mental Health Statistics Improvement Program (MHSIP). IV. Methodology and Survey Administration Two survey populations (sample frames) were identified: 1. Adults - defined as Title XIX/XXI behavioral health recipients who are 18 years of age or older, and are enrolled in any of the adult programs: Serious Mental Illness (SMI) and Drug/Alcohol or General Mental Health (Non-SMI). 2. Youth - defined as Title XIX/XXI behavioral health recipients under age 18 and enrolled in the Child/Adolescent program. A total of 79,443 adult and 39,077 youth RBHA-enrolled Title XIX/XXI consumers were eligible to participate in the 2009 survey. Please refer to Attachment A, 2009 Consumer Survey Protocol, for details on sample frame development, inclusion/exclusion criteria, survey instruments, and survey administration guidelines. V. Data Analysis All completed surveys were entered by each RBHA and submitted in a database to ADHS/DBHS. RBHAs were provided with data files containing the survey responses of their respective consumers. Each RBHA analyzed its respective survey data using an SPSS script that was provided by ADHS/DBHS to ensure consistency in data analysis. Statewide survey data is analyzed as follows: By Domain; By Domain Line Item; Sub-group Analysis; ADHS/DBHS Performance Measure Questions; and Comparison to Past Survey Performance VI. Statewide Survey Data Limitations The following circumstances may have negatively impacted the response rates for the 2009 survey: The survey was originally to be administered April through June 2009. ADHS/DBHS postponed the start date and the survey was conducted May through July. The revised timeframe placed the survey period outside of the school year when, historically, the RBHAs have found the Youth response rate is much better because they tend to have more appointments during the school year, thus increasing the opportunity to obtain a completed survey. It is generally felt that the length of the survey, increased in 2007 with the addition of new MHSIP domains, may discourage clients from participating. Therefore, ADHS/DBHS eliminated several state-added questions from the survey for the 2010 survey year. VII. Survey Findings MHSIP ADULT Demographics A total of 1,711 completed adult surveys were analyzed. Of the adult survey respondents: 63 % were female; 72 % were between the ages of 31-65 years; 5 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 77 % identified their ethnicity as Non Hispanic/Latino; 86 % identified their race as White; 47 % reported receiving behavioral health services for five or more years; and 56 % were enrolled as Seriously Mentally Ill (SMI) and 44% were in Non-SMI programs (General Mental Health or Substance Abuse services). Please see Attachment B, 2009 Adult Demographics, for complete demographics information. Domain Line Item Analysis Line items are specific questions pertaining to each survey domain (see Attachment C, 2009 Adult Line Item). Analysis of answers to each domain specific line item indicates the aspects of service respondents reported as either positively or negatively affecting the overall domain score. The 2009 adult survey findings indicate: General Satisfaction: Respondents report liking the services they receive at their agency; that they would choose their current agency given other choices for service centers; and they would recommend the agency to a friend or family member. Service Access: Respondents report services were available at times that were good for them; felt staff were accessible and that the location of services was convenient. While the overall domain remained under 80%, there was a five percentage point increase in positive responses in this domain from the previous year. Line item analysis indicates improvements can be made in timeliness of agency staff returning clients’ phone calls; informing the consumer of the array of services available to them; and providing more scheduling time to see a psychiatrist. Participation in Treatment Planning: Respondents indicate feeling comfortable asking questions about their treatment and medications; report positively that they, and not staff, decided treatment goals and that they took the lead in deciding their treatment goals. Service Quality and Appropriateness: Respondents positively report receiving information on their rights; feeling that their staff encouraged personal responsibility and empowered growth, change and recovery; that staff were considerate of their cultural background; that they were informed of the side effects of their medications; the staff respected their wishes about who should and should not receive information on their treatment; and they were encouraged to utilize consumer run programs. Outcomes: This domain received a lower overall positive response rate than many of the other adult survey domains (70%). However, 2009 survey results indicate a two percentage point increase from the 2008 survey administration. Areas for improvement include improving respondents’ perceptions of their ability to deal effectively with daily problems; work/school situations, housing and symptoms management. Improved Functioning: Respondents report a lower positive response rate to this outcomes related domain as compared to the aforementioned Outcomes questions. The rate of positive responses to this survey domain is identical to that of the previous year. Improvements can be made around symptom identification, education, and management. Social Connectedness: The rate of positive responses for Social Connectedness domain is identical to those reported in 2008 and mirrors those reported for the Outcomes domain questions. 6 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 Areas for improvement include opportunities to increase social interactions and a sense of belonging and the identification of a natural support system for times of crisis. Summary of 2009 Statewide Results for Adults General Satisfaction Service Access Participation in Treatment Planning Cenpatico-2 86% 81% 82% Cenpatico-4 86% 83% CPSA-3 83% CPSA-5 Service Quality & Appropriatenes s Outcomes Improved Functioning Social Connecte dness 89% 71% 67% 71% 84% 89% 70% 64% 66% 79% 80% 88% 71% 71% 71% 78% 69% 78% 87% 68% 65% 65% Magellan 86% 82% 85% 90% 73% 71% 69% NARBHA 84% 79% 80% 88% 64% 63% 68% Statewide RBHA 84% 78% 82% 89% 70% 68% 68% RBHA RBHA numbers and percentages are based on actual valid survey returns. Statewide RBHA numbers and percentages are based on weighted scores ADHS/DBHS Performance Measures In addition to evaluation of performance in the MHSIP domain areas, ADHS/DBHS assessed four Adult Performance Measures through the Survey in 2009. The Minimum Performance Standard (MPS) for the performance measures is 85%, with the exception of Cultural Competency, which has an MPS of 75%. Statewide results indicate none of these MPS were met. Results of the 2009 Survey Performance Measure questions indicate: Symptomatic Improvement: 63% of RBHA survey respondents reported positively that their symptoms were not bothering them as much. Improvements can be made in symptoms education, identification and self management for behavioral health recipients; and supervision and training of clinicians in the assessment and documentation of symptoms management and target outcomes. Informed Consent: 83% reported positively that their doctor explained their prescribed medication benefits and risks as well as the alternatives to their prescriptions in a way they understood. Member and Family Involvement: 68% reported positively that their families were involved in their treatment. Involvement of family members in adult consumers’ treatment planning and services remains an area for improvement. Cultural Competency: 63% percent reported a positive response to the question designated to measure Cultural Competency, a decrease from the positive response rate 7 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 reported for the similar line item question under the Service Quality and Appropriateness domain that received an 89% positive response rate. Improvements can be made in the inclusion and/or consideration of cultural/race/ethnicity preferences in service planning for adults through training and education of clinical staff and adult consumers. 2009 Adult Consumer Survey, Number and Percentage of Positive Responses to Performance Measure Questions Symptomatic Informed Member/Family Cultural Improvement Consent Involvement Competency MPS: 85% MPS: 85% MPS: 85% MPS: 75% T/RBHA N % N % N % N % Cenpatico-2 179 69% 213 84% 164 70% 157 71% Cenpatico-4 89 63% 107 78% 95 75% 68 61% CPSA 3 160 68% 180 80% 139 68% 121 68% CPSA 5 181 62% 235 81% 171 65% 131 56% Magellan 115 64% 146 84% 104 66% 89 65% NARBHA 163 56% 243 87% 195 77% 146 65% Statewide RBHA 874 63% 1125 83% 838 68% 682 63% T/RBHA numbers are based on actual valid survey returns. Statewide RBHA numbers are based on weighted scores. TRBHA scores are not included in statewide numbers due to TRBHA convenience sampling methodology. Comparison to Past Performance ADHS/DBHS has administered the Consumer Survey since 2001 and began yearly administration in 2005. The following table compares current survey performance across the survey domains to prior administrations. N/A indicates that the domain was not included in the SAMHSA survey that administration year. 2009 Consumer Survey Statewide RBHA Domain Score Trends for Adult Respondents Adult Domain 2001 2003 2005 2006 2007 2008 2009 General Satisfaction 80% 88% 80% 83% 86% 82% 84% Service Access 71% 77% 75% 75% 77% 73% 78% Participation in Treatment Planning N/A 75% 71% 77% 79% 89% 82% Service Quality 79% 88% 84% 84% 88% 84% 89% Outcomes 58% 66% 63% 67% 74% 68% 70% Improved Functioning N/A N/A N/A N/A 66% 68% 68% Social Connectedness N/A N/A N/A N/A 65% 68% 68% 8 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 A review of positive responses attributed to the survey domains across the administration periods indicates that the 2009 survey garnered higher positive response rates for four domains; a lower positive response rate for one domain; and at the 2008 rates for two domains. Adult survey respondents indicated an increase in positive response rates for the domains of General Satisfaction, Service Access, Service Quality and Outcomes in 2009. Rates indicated a 7 percentage point lower positive perception to Participation in Treatment Planning this survey administration; however, these results remain higher than 2003 levels by 7 percentage points. Both Improved Functioning and Social Connectedness remained at 68% positive response rates for each domain in 2009. The Outcomes domain continues an upward trend in positive responses from the inaugural survey administration and improved by 2 percentage points over 2008. MHSIP YSS-F Demographics A total of 1,352 completed youth surveys were analyzed. Of the youth survey respondents: 64 % were male; 92 % were 5-18 years of age; 57 % identified Non Hispanic/Latino as their ethnicity; 81 % identified White as their race; and 53% reported receiving behavioral health services for one to five years. Please see Attachment D, 2009 YSS-F Respondent Demographics, for complete demographic data. Domain Line Item Analysis Line items are specific questions pertaining to each survey domain. Analysis of answers to each domain specific line item indicates the aspects of service delivery the respondents report as either positively or negatively affecting the overall domain score (Attachment E, 2009 YSS-F Line Item). 2009 YSS-F survey findings indicate: General Satisfaction: Respondents report satisfaction with their child’s services; feeling continuously supported by staff; and that the services they receive are appropriate for their family. Service Access: Families felt that both the location and times allocated for service provision were convenient. Participation in Treatment Planning: The majority of respondents indicated participating in their child’s treatment by choosing both their child’s services and treatment goals. Cultural Sensitivity: Families indicated that staff spoke to them in a manner in which they understood, treated them with respect and respected their unique cultural/religious/spiritual beliefs and norms. Outcomes: This domain received a positive response rate of 68%, an improvement of 8 percentage points over 2008 rates. Improvements can be made in educating and supporting families in coping skills training and development to increase satisfaction with family life and symptom management. Improved Functioning: Respondents reported the same response rate for this domain as for the Outcomes domain (68%); however, this rate indicates an increase of 8 percentage points from the previous year’s administration. 9 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 As indicated for the Outcomes domain, improvements can be made in educating and supporting families in coping skills training and development to increase satisfaction with family life and symptom management. Social Connectedness: Findings indicate that families feel comfortable talking with supports about their child’s problems; the families have support systems available to them for crisis situations and opportunities for social activities. Positive response rates for this outcomes related domain received higher response rates than that of both Outcomes and Improved Functioning at 83%. Summary of 2009 Statewide Results for Youth General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness Cenpatico-2 81% 82% 94% 93% 62% 61% 85% Cenpatico-4 85% 91% 98% 96% 66% 65% 87% CPSA-3 75% 76% 88% 89% 66% 65% 78% CPSA-5 82% 74% 86% 86% 72% 72% 84% Magellan 77% 73% 88% 91% 68% 64% 82% NARBHA 83% 79% 92% 93% 67% 66% 87% Statewide RBHA 80% 76% 89% 90% 68% 66% 83% RBHA T/RBHA numbers and percentages are based on actual valid survey returns. Statewide RBHA numbers and percentages are based on weighted scores. TRBHA scores are not included in statewide numbers due to TRBHA convenience sampling methodology. ADHS/DBHS Performance Measures In addition to the MHSIP questions, ADHS/DBHS has designated three Performance Measures questions to the 2009 YSS-F Survey. The Minimum Performance Standards (MPS) attributed to these measures are 85%, with the exception of Cultural Sensitivity, which has an MPS of 75%. The performance measures apply to the RBHAs only. Findings indicate: Cultural Competency: 87% of YSS-F survey respondents reported positively that staff were sensitive to their cultural/ethnic background. Cultural Competency is the only Performance Measure where the MPS was met statewide. Symptomatic Improvement: 62% of respondents indicated that their child’s symptoms were not bothering their child as much. As with adults, improvements can be made to families in symptom education, identification and self management as well as clinician assessment and documentation of symptom improvements. Informed Consent: 80% reported positively that their child’s doctor explained the benefits, risks and alternatives to the medications prescribed to their child and they understood the information as presented. 10 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 2009 YSS-F, Number and Percentage of Positive Responses to Performance Measure Questions Cultural Symptomatic Informed Competency Improvement Consent MPS: 75% MPS: 85% MPS: 85% T/RBHA N % N % N % Cenpatico-2 145 90% 91 59% 121 86% Cenpatico-4 106 94% 78 65% 88 87% CPSA 3 155 88% 117 64% 132 78% CPSA 5 165 83% 112 58% 132 79% Magellan 133 87% 103 64% 114 78% NARBHA Statewide RBHA 255 89% 181 58% 213 82% 931 87% 686 62% 781 80% RBHA numbers are based on actual valid survey returns. Statewide RBHA numbers are based on weighted scores. Comparison to Past Performance ADHS/DBHS has administered the Consumer Survey since 2001 and began yearly administration in 2005. The following table compares current survey performance across the survey domains to prior administrations. N/A indicates that the domain was not included in the SAMHSA survey that administration year. 2009 Consumer Survey Statewide RBHA Domain Score Trends for YSS-F Respondents YSS-F Domain 2001 2003 2005 2006 2007 2008 2009 General Satisfaction Service Access Participation in Txt Planning Service Quality Outcomes Improved Functioning Social Connectedness 68% 70% N/A N/A 51% N/A N/A 80% 78% 85% 93% 62% N/A N/A 74% 72% 84% 92% 60% N/A N/A 78% 75% 87% 94% 62% N/A N/A 78% 78% 90% 92% 58% 61% 82% 77% 73% 87% 90% 60% 58% 79% 80% 76% 89% 90% 68% 66% 83% Families reported higher rates of satisfaction across six of the seven YSS-F domains this year as compared to 2008, with the exception of Service Quality and Appropriateness, which remained at the 2008 rates of 90%. Outcomes and Improved Functioning increased by 8 percentage points from 2008 rates but as seen with the adult respondents, the Outcomes and Improved Functioning domains remain the lowest rates of satisfaction. 11 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 VI. Statewide Improvement Opportunities ADHS/DBHS recognizes the need to identify targeted improvement efforts supported by the 2009 survey results. The following are ADHS/DBHS that address survey findings: Outcomes – The survey data indicates improvements can be made in consumers’ perception and understanding of treatment and functional outcomes, as well as front line clinical staff assisting the consumers with identification and tracking of identified outcomes. 2009 Adult Consumer Survey Trends Outcomes 2001 - 2009 Positive Response 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 74% 66% 63% 67% 2003 2005 2006 58% 2001 70% 2008 2009 Adult Domain 2001 2003 2005 2006 2007 2008 2009 Outcomes 58% 66% 63% 67% 74% 68% 70% 2009 YSS-F Consumer Survey Trends Outcomes 2001 - 2009 Positive Response 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 62% 60% 62% 58% 60% 2003 2005 2006 2007 2008 51% 2001 12 2007 68% YSS-F Domain 2001 2003 2005 2006 2007 2008 2009 Outcomes 51% 62% 60% 62% 58% 60% 68% Division of Quality Management Operations Office of Performance Improvement 68% 2009 2009 Consumer Survey Report July 30, 2010 Below are ADHS/DBHS initiatives that address outcomes:  ADHS/DBHS is currently using the Children’s System of Care Practice Review (CSOCPR) beginning in early Fiscal Year 2010 (FY10). This revised review focuses on Child and Family Teams (CFTs) and includes outcomes focused questions to ensure the CFTs are identifying and tracking outcomes as well as evaluating the effectiveness of provided services. The CSOCPR provides valuable data for ADHS/DBHS and the RBHAs pertaining to children’s outcomes.  The ADHS/DBHS Adult System of Care Plan identifies several goals targeting outcomes such as: increasing employment, increasing peer and family supports, and the adoption and dissemination of adult recovery principles.  ADHS/DBHS, along with family member and recipient input, is developing the ADHS/DBHS Outcomes Framework as a means to direct resources and develop initiatives based on family and recipient identified outcomes, the National Outcomes Measures (NOMs) and ADHS/DBHS performance metrics. The development of the Outcomes Framework is in direct response to the identified need for improved family, recipient and provider education and tracking of outcomes as a means to drive both individual recipient and program outcomes. Access to Services - Survey data indicates improvements can be made in consumers’ perception of ease of access to, and availability of, psychiatric and support services; and increased accessibility of support staff.  ADHS/DBHS is implementing a new enrollment process that automatically provides any Arizona Health Care Cost Containment System (AHCCCS) member with a home RBHA based on area of residence. AHCCCS and ADHS/DBHS are implementing this process to eliminate potential administrative barriers to accessing behavioral health services in the publicly funded healthcare system. VII. Conclusion 2009 findings indicate increases in overall positive response rates across the domains of both the Adult and YSS-F surveys from the previous survey administration. However, adults reported decreased involvement in their service planning and showed no change in their perceptions of improved functioning and social connectedness. The families of child behavioral health recipients feel supported by assigned staff, believe they are able to participate in their child’s treatment planning, feel their cultural needs are assessed and respected and feel supported by family and friends. Similar to findings across survey periods, outcomes identification, tracking and monitoring continues to be an area for improvement, as well as behavioral health recipients feeling they can access the services they want, when they want. ADHS/DBHS is committed to improving the tracking and utilization of outcomes data. Outcomes data is used by ADHS/DBHS to support quality initiatives, assess the viability of existing programming and improve services delivered to persons served in the ADHS/DBHS system of care. ADHS/DBHS believes that removing barriers to accessing available services and 13 Division of Quality Management Operations Office of Performance Improvement 2009 Consumer Survey Report July 30, 2010 assisting members in the identification and tracking of treatment outcomes is imperative to improving behavioral health recipients’ overall perception of the quality of services they receive. 14 Division of Quality Management Operations Office of Performance Improvement