2010 Annual Consumer Survey Report June, 2011 Revised September, 2011 Jennifer Tonges, Quality Management Specialist Jay Dunkleberger, Manager Julie Karcis, Consultant ADHS/DBHS Office of Performance Improvement 2010 Consumer Survey Report June, 2011 TABLE OF CONTENTS EXECUTIVE SUMMARY……………………………………………………………………………… ............ 3 INTRODUCTION………………………………………………………………………………...5 SURVEY RESPONSE RATES .…………………………………………………………… ........ 5 METHODOLOGY AND SURVEY ADMINISTRATION……………………………………………………………………………...6 DATA ANALYSIS………………….………………………………………………………… .... 6 SURVEY FINDINGS................................................................ ..................................................... 7 STATEWIDE IMPROVEMENT OPPORTUNITIES……………………………………………… ............................................... 11 CONCLUSION…………………………………………………………………………………..14 ATTACHMENTS Attachment A: 2010 Tribal Behavioral Health Survey Results Attachment B: 2010 Consumer Survey Protocol Attachment C: 2010 Raw Data Attachment D: 2010 Adult Demographics Attachment E: 2010 Adult Line Item Attachment F: 2010 YSS-F Demographics Attachment G: 2010 YSS-F Line Item Attachment H: List of Acronyms APPENDICES YEAR 2010 ADULT CONSUMER SURVEY (ENGLISH AND SPANISH) YEAR 2010 YOUTH CONSUMER SURVEY FOR FAMILIES (ENGLISH AND SPANISH) 2010 Consumer Survey Report September, 2011 I. Executive Summary The Arizona Department of Health Services/Division of Behavioral Health Services (ADHS/DBHS), Tribal/Regional Behavioral Health Authorities (T/RBHAs) and contracted service providers jointly conduct statewide consumer surveys during each fiscal year. Each year, two surveys are administered based on the Substance Abuse and Mental Health Services Administration‟s (SAMSHA‟s) Mental Health Statistics Improvement Program (MHSIP) consumer surveys: The Adult Consumer Survey and The Youth Services Survey for Families (YSS-F). The surveys request independent feedback from Title XIX/XXI adults and families of youth receiving services through Arizona‟s publicly funded behavioral health system. The surveys measure consumers‟ perceptions of behavioral health services in relation to the following domains: General Satisfaction Access to Services Service Quality/Appropriateness Participation in Treatment Outcomes Cultural Sensitivity Improved Functioning Social Connectedness ADHS/DBHS Consumer Survey data results are routinely:  Reviewed in the ADHS/DBHS Quality Management (QM) Committee meetings;  Used as the basis for recommendations from the QM Committee, which are incorporated in RBHA contracts and QM Plans;  Disseminated to ADHS/DBHS stakeholders and consumers via the ADHS/DBHS website. Survey outcomes are always reported to:  The National Association of State Mental Health Program Directors‟ (NASMHPD)  National Research Institute (NRI)  Western States Decision Support Group (WSDSG)  Substance Abuse and Mental Health Services Administration‟s (SAMHSA) Mental Health Statistics Improvement Program (MHSIP) The fiscal year 2010 survey was conducted from April 1 through May 28, 2010. The overall findings of improvement in the Adult Consumer Survey and YSS-F include: All adult and youth domains showed improvement from 2009; Three adult domains, General Satisfaction, Participation in Treatment Planning and Service Quality and Appropriateness, scored over 85 percent; The adult survey results, with the exception of the General Satisfaction domain, received the highest satisfaction rate compared to previous survey results; Four youth domains, General Satisfaction, Participation in Treatment Planning, Cultural Sensitivity and Social Connectedness, scored over 85 percent; 3 2010 Consumer Survey Report September, 2011 Statewide youth results were the highest that they have been since the survey began or the domain has been measured. Comparison between 2009 and 2010 Adult Consumer Survey statewide results shows a statistically significant 11 percent increase in Social Connectedness (P = 0.02) and a 10 percent increase in Participation in Treatment Planning (P = 0.01). In the YSS-F Survey, there was a statistically significant six (6) percent statewide increase in Cultural Sensitivity (P = 0.05). No domains in either survey decreased from 2009 to 2010. ADHS/DBHS intends to pursue initiatives with the RBHAs to continue to improve scores for each domain. RBHA and TRBHA results from the 2010 Consumer Survey are presented separately to preserve the integrity of findings as two different sampling methodologies were used to get the survey sample. (For details, please refer to Section III-Survey Response Rates.) 4 2010 Consumer Survey Report September, 2011 II. Introduction The Arizona Department of Health Services/Division of Behavioral Health Services (ADHS/DBHS) and the Regional Behavioral Health Authorities (RBHAs) in collaboration with their contracted providers administered the statewide consumer survey during April and May of 2010. Survey data results are:  Reviewed in the ADHS/DBHS Quality Management (QM) Committee meetings;  Used as the basis for recommendations from the QM Committee, which are incorporated in RBHA contracts and QM Plans;  Disseminated to ADHS/DBHS stakeholders and consumers via the ADHS/DBHS website. Survey outcomes are reported to:  The National Association of State Mental Health Program Directors‟ (NASMHPD)  National Research Institute (NRI)  Western States Decision Support Group (WSDSG)  Substance Abuse and Mental Health Services Administration‟s (SAMHSA) Mental Health Statistics Improvement Program (MHSIP) III. Survey Response Rates A total of 5,800 RBHA-enrolled consumers were included for the MHSIP Consumer Survey random sample pool; 4,179 consumers were contacted and asked to complete a survey. The response rate is calculated by dividing the number of surveys returned by the number of enrollees asked to participate. Statewide response rates based on surveys offered were 78% for adults and 73% for youth, as seen in TABLE-1. Please see Attachment B for a complete discussion of the survey protocol. The TRBHA scores are not included in the statewide results due to the TRBHA convenience sampling methodology which solicits responses at the convenience of the recipients rather than from a random sample of TRBHA enrolled recipients. Therefore the results are reported separately in Attachment A-2010 Tribal Regional Behavioral Health Authority Results. TABLE-1: Survey Response Rate Based on Surveys Offered for Each RBHA/GSA Consumer Survey 2010 Surveys Offered (a) Surveys Completed (b) Response Rate (b/a) RBHA/GSA Overall Adult Youth Overall Adult Youth Overall Adult Youth CBHS 2 CBHS 4 CPSA 3 CPSA 5 Magellan NARBHA Statewide RBHA 5 662 714 698 712 600 793 355 356 382 367 313 377 307 358 316 345 287 416 4,179 2,150 2,029 305 536 557 585 495 695 189 267 305 327 270 324 116 269 252 258 225 371 46% 75% 80% 82% 83% 88% 53% 75% 80% 89% 86% 86% 38% 75% 80% 75% 78% 89% 3,173 1,682 1,491 76% 78% 73% 2010 Consumer Survey Report September, 2011 IV. Methodology and Survey Administration Two survey populations (sampling frame) were identified leading to the use of the Adult Consumer Survey and the YSS-F: Adults - defined as Title XIX/XXI behavioral health recipients who are 18 years of age or older, and are enrolled in any of the adult programs: Serious Mental Illness (SMI) and Drug/Alcohol or General Mental Health (Non-SMI). Youth - defined as Title XIX/XXI behavioral health recipients under age 18 and enrolled in the Child/Adolescent program. A total of 84,119 adult and 41,585 youth RBHA-enrolled Title XIX/XXI consumers were eligible to participate in the 2010 survey. Please refer to Attachment B (2010 Consumer Survey Protocol) for details on sampling frame development, inclusion/exclusion criteria, survey instruments, and survey administration guidelines. The ADHS/DBHS Consumer Survey utilized a Likert Scale model to have respondents rate the domain questions from strongly agree (numeric value of 1) to strongly disagree (numeric value of 5), and an option for Not Applicable (numeric value 0). Consumer perception was determined as positive for those domains where the question responses averaged 2.5 or below. TABLE-2 and TABLE-3 show the number and percentage of sample surveys that were not included in the domain score, presented by RBHA/GSA and domain. Surveys are excluded from a domain‟s score when one-third or more of the questions have a response that cannot be determined. This occurs most often when no response is marked, and can also occur if more than one response is marked. See Attachment B for a full discussion of the survey protocol. TABLE-2: Adult Surveys for which responses could not be determined RBHA General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness Cenpatico-2 Cenpatico-4 CPSA-3 CPSA-5 Magellan NARBHA Statewide RBHA 8 (4.2%) 10 (3.7%) 22 (7.2%) 22 (6.7%) 27 (10.0%) 8 (2.5%) 6 (3.2%) 5 (1.9%) 20 (6.6%) 22 (6.7%) 19 (7.0%) 9 (2.8%) 11 (5.8%) 14 (5.2%) 29 (9.5%) 39 (11.9%) 39 (14.4%) 7 (2.2%) 8 (4.2%) 9 (3.4%) 22 (7.2%) 28 (8.6%) 21 (7.8%) 4 (1.2%) 16 (8.5%) 11 (4.1%) 25 (8.2%) 33 (10.1%) 22 (8.1%) 13 (4.0%) 12 (6.3%) 7 (2.6%) 23 (7.5%) 35 (10.7%) 26 (9.6%) 9 (2.8%) 6 (3.2%) 7 (2.6%) 22 (7.2%) 23 (7.0%) 19 (7.0%) 7 (2.2%) 97 (5.8%) 81 (4.8%) 139 (8.3%) 92 (5.4%) 120 (7.1%) 112 (6.6%) 84 (5.0%) 6 2010 Consumer Survey Report September, 2011 TABLE-3: Youth Surveys for which responses could not be determined RBHA Cenpatico-2 Cenpatico-4 CPSA-3 CPSA-5 Magellan NARBHA Statewide RBHA General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness 5 (4.4%) 9 (3.5%) 7 (2.8%) 7 (2.7%) 15 (6.7%) 8 (2.2%) 1 (0.9%) 13 (5.0%) 6 (2.4%) 9 (3.5%) 24 (10.7%) 8 (2.2%) 5 (4.4%) 11 (4.3%) 16 (6.3%) 17 (6.6%) 26 (11.6%) 19 (5.2%) 3 (2.6%) 11 (4.3%) 5 (2.0%) 19 (7.4%) 21 (9.3%) 29 (7.9%) 4 (3.5%) 13 (5.0%) 8 (3.2%) 8 (3.1%) 12 (5.3%) 13 (3.5%) 6 (5.3%) 14 (5.4%) 9 (3.6%) 9 (3.5%) 16 (7.1%) 16 (4.3%) 6 (5.3%) 15 (5.8%) 5 (2.0%) 17 (6.6%) 16 (7.1%) 8 (2.2%) 51 (3.4%) 61 (4.1%) 94 (6.4%) 88 (6.0%) 58 (3.9%) 70 (4.7%) 67 (4.5%) V. Data Analysis ADHS/DBHS provided each RBHA with an Access database to compile survey responses. All completed surveys were entered and the database was submitted to ADHS/DBHS. Each RBHA analyzed its respective survey data using an SPSS script designed and provided by ADHS/DBHS to ensure consistency in data analysis. Statewide survey data is analyzed as follows: By Domain; By Domain Line Item; Sub-group Analysis; ADHS/DBHS Performance Measure Questions; and Comparison to Past Survey Performance. The statewide survey results are weighted to correct the sample design, taking the different population sizes for each Geographic Service Area (GSA) into consideration. The weighted averages were taken to avoid the over/under representation of highly/less populated GSAs. Please refer to Attachment C, 2010 Raw Data, for further specifics of the weighting methodology and weights applied to each GSA. VI. Statewide Survey Data Limitations Two different sampling methodologies were used to get the 2010 Consumer Survey sample. The TRBHA scores could not be included in the statewide results due to the TRBHA convenience sampling methodology which solicits responses at the convenience of the recipients and providers rather than from a random sample of TRBHA enrolled recipients. 7 2010 Consumer Survey Report September, 2011 VII. Survey Findings MHSIP ADULT Demographics A total of 1,682 completed adult surveys were analyzed. The breakout of the respondents is by gender, age, ethnicity, race, length of enrollment in services, length of enrollment and behavioral health category. Please see Attachment D, 2010 Adult Demographics for complete demographics information. Domain Line Item Analysis A specific set of questions was developed to gather scores for each domain. Line items are specific questions pertaining to each survey domain (Attachment E, 2010 Adult Line Item); analysis of the respondents‟ answers to a question under a domain, either positive or negative, affects the overall domain score. The 2010 adult survey findings identify: General Satisfaction: Three questions were asked in the survey to ascertain General Satisfaction (Attachment E). The overall score for this domain was 86% which is an average score based on the 3 questions. Service Access: Six questions in the survey gathered information regarding Access to Services (Attachment E). This domain scored at 81%, based on an average score of six (6) questions. Although there was a 3% increase on Service Access from 2009, ADHS/DBHS will pursue initiatives with the RBHAs to continue to improve scores. Line item analysis indicates improvements can be made in timeliness of agency staff returning clients‟ phone calls and providing more schedule availability for psychiatrists. Participation in Treatment Planning: The survey comprised two questions to indicate Participation in Treatment Planning (Attachment E). The overall score for this domain was 92%. There was a significant increase in the domain score from 2009 (P <= 0.01). Service Quality and Appropriateness: Nine questions in the survey focused on Service Quality and Appropriateness (Attachment E). The overall score for this domain was 90%, representing an average score based on nine (9) questions. Outcomes: Eight questions were asked to get consumers‟ perception on Outcomes (Attachment E). The overall score for this domain was 75%. This indicates a 5% increase in Outcomes compared to 2009, however the increase was not statistically significant. ADHS/DBHS will continue to focus on improvements and initiatives in Outcomes. Areas to focus on improving are respondents‟ perception of situations in work/school, housing, and symptom management. Improved Functioning: Five questions indicate respondent‟s impressions of Improved Functioning (Attachment E). The overall score for this domain was 71%. Although Improved Functioning scores increased by 3% since 2009, the increase was not statistically significant. ADHS/DBHS will continue to implement initiatives to improve this domain. 8 2010 Consumer Survey Report September, 2011 Social Connectedness: Four questions were asked in the survey to ascertain Social Connectedness (Attachment E). The score for this domain was 79%. Social Connectedness scores had an 11% increase from 2009, which was statistically significant (P = 0.02). This domain will remain a focus for improvement for ADHS/DBHS. Areas for improvement include opportunities to increase a sense of belonging in the community. TABLE-4 through TABLE-9 show positive response rates for each RBHA/GSA by domain. Included are the years 2007 through 2010. A P-value of 95% significance when less than or equal to 0.05 or “ns” for “not significant” is listed to reflect the change from the prior year. TABLE-10 shows statewide RBHA results for those years. The statewide statistics are based on weighted scores. TABLE-4: Cenpatico-2 Adult Consumer Survey Results by Domain for 2007 - 2010 2007 % Domain General Satisfaction % 2008 (P-value) 2009 % (P-value) 2010 % (P-value) 87 84 (ns) 86 (ns) 91 (ns) 80 82 (ns) 81 (ns) 90 (0.02) 79 86 (ns) 82 (ns) 93 (0.00) 87 88 (ns) 89 (ns) 92 (ns) Outcomes 78 71 (ns) 71 (ns) 81 (0.03) Improved Functioning 73 75 (ns) 67 (ns) 75 (ns) Social Connectedness 74 75 (ns) 71 (ns) 83 (0.01) Service Access Participation in Treatment Planning Service Quality & Appropriateness TABLE-5: Cenpatico-4 Adult Consumer Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness Color 2007 % 87 86 2008 (P-value) 87 (ns) 83 (ns) 2009 % (P-value) 86 (ns) 83 (ns) 2010 % (P-value) 87 (ns) 87 (ns) 81 92 (0.01) 84 (0.00) 92 (0.03) 91 78 76 73 90 73 67 67 (ns) (ns) (0.04) (ns) 89 70 64 66 (ns) (ns) (ns) (ns) 90 75 73 77 (ns) (ns) (ns) (0.02) Key statistically significant increase statistically significant decrease 9 % 2010 Consumer Survey Report September, 2011 TABLE-6: CPSA-3 Adult Consumer Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness 2007 % 90 83 % 2008 (P-value) 82 (0.01) 77 (ns) 2009 % (P-value) 83 (ns) 79 (ns) 2010 % (P-value) 86 (ns) 84 (ns) 82 87 (ns) 79 (0.02) 88 (0.03) 89 79 72 71 86 68 73 72 (ns) (0.01) (ns) (ns) 88 71 71 71 (ns) (ns) (ns) (ns) 88 77 72 80 (ns) (ns) (ns) (0.05) TABLE-7: CPSA-5 Adult Consumer Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness 2007 % 85 78 % 2008 (P-value) 83 (ns) 67 (0.01) 2009 % (P-value) 78 (ns) 69 (ns) 2010 % (P-value) 87 (0.03) 79 (0.03) 78 91 (0.00) 78 (0.00) 92 (0.00) 89 70 64 64 80 66 64 62 (0.00) (ns) (ns) (ns) 87 68 65 65 (ns) (ns) (ns) (ns) 90 69 68 78 (ns) (ns) (ns) (0.01) TABLE-8: Adult Consumer Survey Results by Domain for Value Options in 2007 and Magellan in 2008-2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness Color 2007 % 87 83 2008 (P-value) 80 (0.04) 71 (0.00) 2009 % (P-value) 86 (ns) 82 (0.02) 2010 % (P-value) 85 (ns) 79 (ns) 83 87 (ns) 85 (ns) 92 (0.05) 90 77 67 67 83 68 68 69 (0.02) (0.03) (ns) (ns) 90 73 71 69 (ns) (ns) (ns) (ns) 90 77 72 79 (ns) (ns) (ns) (0.03) Key statistically significant increase statistically significant decrease 10 % 2010 Consumer Survey Report September, 2011 TABLE-9: NARBHA Adult Consumer Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness 2007 % 84 73 % 2008 (P-value) 85 (ns) 80 (ns) 2009 % (P-value) 84 (ns) 79 (ns) 2010 % (P-value) 88 (ns) 83 (ns) 78 92 (0.00) 80 (0.00) 92 (0.00) 87 74 63 61 88 71 68 65 (ns) (ns) (ns) (ns) 88 64 63 68 (ns) (ns) (ns) (ns) 90 72 71 78 (ns) (ns) (ns) (0.03) TABLE-10 Statewide RBHA Adult Consumer Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness Color 2007 % 86 77 % 2008 (P-value) 82 (ns) 73 (ns) 2009 % (P-value) 84 (ns) 78 (ns) 2010 % (P-value) 86 (ns) 81 (ns) 79 89 (0.01) 82 (0.03) 92 (0.01) 88 74 66 65 84 68 68 68 (ns) (ns) (ns) (ns) 89 70 68 68 (ns) (ns) (ns) (ns) 90 75 71 79 (ns) (ns) (ns) (0.02) Key statistically significant increase statistically significant decrease 2010 statewide RBHA adult survey results show the highest positive increase in responses since implementation of the survey. The improvements made in Social Connectedness and Participation in Treatment Planning domains were among the highest between any survey administration periods. Social Connectedness improved significantly from 2009 making it the largest improvement of any domain in all of the administrative periods. Participation in Treatment Planning jumped 10% from 2009. These increases were statistically significant; there were no statistically significant decreases in any domains for the RBHAs or statewide for this survey administration. MHSIP YSS-F Demographics A total of 1,491 completed youth surveys were analyzed. The breakout of the respondents is reported by gender, age group, ethnicity, race, length of enrollment in services, and involvement in a Child and Family Team (CFT). Please see Attachment F, 2010 YSS-F Demographics for complete demographics information. 11 2010 Consumer Survey Report September, 2011 Domain Line Item Analysis Line items are specific questions pertaining to each survey domain. Analysis of answers to each domain specific line item indicates the aspects of service delivery the respondents report as either positively or negatively affecting the overall domain score (Attachment G, 2010 YSS-F Line Item). 2010 YSS-F survey findings indicate: General Satisfaction: Six questions were asked in the survey to ascertain General Satisfaction (Attachment G). The overall score for this domain was 85% based on the average score of 6 questions. Service Access: Two questions in the survey gathered information regarding Access to Services (Attachment G). This domain scored 83%, representing an average score based on 2 questions. ADHS/DBHS continues to implement initiatives to improve scores in this domain. Participation in Treatment Planning: The survey comprised three questions to indicate Participation in Treatment Planning (see Attachment G). The score for this domain was 92%. Cultural Sensitivity: Four questions in the survey focused on Cultural Sensitivity (Attachment G). This domain scored 96%, highest score achieved by any domain, a statistically significant increase (P = 0.05). Outcomes: Eight questions were asked to get consumers‟ perception on Outcomes (Attachment G). The overall score for this domain was 69%; indicating just one percent increase from 2009. ADHS/DBHS will continue to focus on implementing initiatives to improve this domain. Initiatives will focus on how the child handles daily life, positive relationships with family members and performance in school and/or work. Improved Functioning: Six questions indicate respondent‟s impressions of Improved Functioning (Attachment G). The overall score for this domain was 71%. This domain increased by five (5) percent from 2009, however this increase was not statistically significant. ADHS/DBHS will continue to implement initiatives to improve this domain. Social Connectedness: Four questions were asked in the survey to ascertain Social Connectedness (see Attachment G). This domain scored 88%. TABLE-11 through TABLE-16 show results for each RBHA/GSA by domain. Included are the years 2007 through 2010. A P-value of 95% significance when less than or equal to 0.05 or “ns” for “not significant” is listed to reflect the change from the prior year. TABLE-17 shows statewide RBHA results for those years. The statewide statistics are based on weighted scores. 12 2010 Consumer Survey Report September, 2011 TABLE-11: Cenpatico-2 Youth Consumer YSS-F Survey Results by Domain for 2007 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness 2007 % 89 86 % 88 96 76 78 89 2008 (P-value) 76 (0.00) 81 (ns) 86 94 70 69 86 (ns) (ns) (ns) (0.03) (ns) 2009 % (P-value) 81 (ns) 82 (ns) 94 93 62 61 85 (0.02) (ns) (ns) (ns) (ns) 2010 % (P-value) 82 (ns) 90 (0.04) 93 99 69 70 89 (ns) (0.02) (ns) (ns) (ns) TABLE-12: Cenpatico-4 Youth Consumer YSS-F Survey Results by Domain for 2007 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness 2007 % 78 86 % 92 91 66 68 82 2008 (P-value) 78 (ns) 80 (ns) 90 92 61 62 87 (ns) (ns) (ns) (ns) (ns) 2009 % (P-value) 85 (ns) 91 (0.01) 98 96 66 65 87 (0.01) (ns) (ns) (ns) (ns) 2010 % (P-value) 80 (ns) 87 (ns) 91 95 66 69 88 (0.00) (ns) (ns) (ns) (ns) TABLE-13: CPSA-3 Youth Consumer YSS-F Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness Color 2007 % 77 74 88 90 65 65 77 Key statistically significant increase statistically significant decrease 13 % 2008 (P-value) 74 (ns) 74 (ns) 80 89 65 62 77 (0.01) (ns) (ns) (ns) (ns) 2009 % (P-value) 75 (ns) 76 (ns) 88 89 66 65 78 (0.05) (ns) (ns) (ns) (ns) 2010 % (P-value) 87 (0.01) 90 (0.00) 90 94 77 79 87 (ns) (ns) (0.02) (0.00) (0.03) 2010 Consumer Survey Report September, 2011 TABLE-14: CPSA-5 Youth Consumer YSS-F Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness 2007 % 81 76 % 89 90 65 66 86 2008 (P-value) 79 (ns) 73 (ns) 86 90 63 62 82 (ns) (ns) (ns) (ns) (ns) 2009 % (P-value) 82 (ns) 74 (ns) 86 86 72 72 84 (ns) (ns) (ns) (0.04) (ns) 2010 % (P-value) 81 (ns) 84 (0.02) 88 94 62 66 84 (ns) (0.02) (0.03) (ns) (ns) TABLE-15: Youth Consumer YSS-F Survey Results by Domain for Value Options in 2007 and Magellan in 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness 2007 % 77 75 % 86 93 62 63 79 2008 (P-value) 75 (ns) 70 (ns) 87 90 56 53 75 (ns) (ns) (ns) (0.04) (ns) 2009 % (P-value) 77 (ns) 73 (ns) 88 91 68 64 82 (ns) (ns) (0.02) (0.03) (ns) 2010 % (P-value) 86 (0.03) 80 (ns) 93 98 71 72 88 (ns) (0.01) (ns) (ns) (ns) TABLE-16: NARBHA Youth Consumer YSS-F Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness Color 2007 % 76 77 91 93 51 55 80 Key statistically significant increase statistically significant decrease 14 % 2008 (P-value) 84 (ns) 81 (ns) 88 93 72 71 87 (ns) (ns) (0.00) (0.00) (ns) 2009 % (P-value) 83 (ns) 79 (ns) 92 93 67 66 87 (ns) (ns) (ns) (ns) (ns) 2010 % (P-value) 90 (ns) 90 (0.01) 93 96 70 73 91 (ns) (ns) (ns) (ns) (ns) 2010 Consumer Survey Report September, 2011 TABLE-17: Statewide Youth Consumer YSS-F Survey Results by Domain for 2007 - 2010 Domain General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness Color 2007 % 78 78 90 92 58 61 82 % 2008 (P-value) 77 (ns) 73 (ns) 87 90 60 58 79 (ns) (ns) (ns) (ns) (ns) 2009 % (P-value) 80 (ns) 76 (ns) 89 90 68 66 83 (ns) (ns) (ns) (ns) (ns) 2010 % (P-value) 85 (ns) 83 (ns) 92 96 69 71 88 (ns) (0.05) (ns) (ns) (ns) Key statistically significant increase statistically significant decrease Families of child behavioral health recipients reported higher positive perceptions (responses) in all of the measured domains when compared to 2009. Each domain is at the highest reported percent since the implementation of the survey. The Cultural Sensitivity domain had a statistically significant increase of 6% (P = 0.05). As with the adult survey, the Outcomes domain remains the lowest, directly affecting consumers‟ perceptions regarding the Improved Functioning domain (six out of seven questions in the Outcomes domain are the same in the Improved Functioning domain-Attachment G). Cenpatico-4 families experienced a statistically significant decrease in Participation in Treatment Planning, as did CPSA-5 families in the Outcomes domain. VIII. Statewide Improvement Opportunities ADHS/DBHS recognizes the opportunity to improve efforts supported by the 2010 survey results; ADHS/DBHS intends to pursue initiatives with the RBHAs to continue to improve scores for each domain. Special attention will be placed around the Outcomes, Improved Functioning, Service Access, and Social Connectedness domains. In addition to general statewide initiatives, RBHA-specific activities are undertaken as part of standard ADHS/DBHS processes, as appropriate. Survey results are shared with the RBHAs, and they are required to respond with analysis of any problem areas, along with plans for improvement. ADHS/DBHS reviews their conclusions and plans, and monitors implementation and progress. The following graphs reflect Adult and Youth survey trends for key areas of focus. 15 2010 Consumer Survey Report September, 2011 Adult Responses Percent of Positive Responses to Service Access, Outcomes, Improved Functioning, and Social Connectedness Domains 2007 -2010 100% % Positive Response 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 2007 2008 2009 2010 Service Access 77% 73% 78% 81% Outcomes 74% 68% 70% 75% Improved Functioning 66% 68% 68% 71% Social Connectedness 65% 68% 68% 79% Service Access Outcomes Improved Functioning Social Connectedness Youth Responses Percent of Positive Responses to Service Access, Outcomes, and Improved Functioning Domains 2007 -2010 100% % Positive Response 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 2007 2008 2009 2010 Service Access 78% 73% 76% 83% Outcomes 58% 60% 68% 69% Improved Functioning 61% 58% 66% 71% Service Access 16 Outcomes Improved Functioning 2010 Consumer Survey Report September, 2011 ADHS/DBHS Initiatives: Below are ADHS/DBHS initiatives for both the Children and Adult systems of care that address Outcomes, Improved Functioning, Services Access and Social Connectedness: Adult Initiatives:  The 2012-2014 ADHS/DBHS Adult System of Care (ASOC)-Strategic Plan (SP) establishes goals and objectives for statewide network development priorities in part through a review of State and GSA network analysis, review of data sets, community input, and the identification of best practices efforts that are designed to assist individuals moving through recovery and increasing the quality of life outcomes. The analyses of the ASOC-strategic plan and the 2010 Adult and Youth Consumer Survey identified similar priorities. For example: o The ASOC-SP includes an objective to increase the percentage of peers and family members satisfied with their access to services which was identified as an area needing improvement in the Consumer Survey Report (CSR). Additionally, ASOCSP includes strategies to conduct statewide discussion on the awareness and community needs around Trauma Informed Care. o Another objective in the ASOC-SP is to increase employment rate for persons with Serious Mental Illness (SMI), General Mental Health (GMH) and Substance Abuse (SA). The Outcomes domain of the Consumer Survey addresses a person‟s ability to: 1) control his/her life, 2) operate in school, and work environment, and 3) do better in social situations; by achieving the improvement in employment rate in ASOC-SP, ADHS/DBHS expects an increase in Outcomes domain.  ADHS/DBHS is in the process of finalizing policies on clinical supervision and have developed a Clinical Supervision Review Tool for T/RBHAs to ensure sufficient clinical supervision for their behavioral health staff, at least four hours monthly. The aim is to improve delivery of services and quality of care which will ultimately affect outcomes and access to care. System Initiatives: 17  ADHS/DBHS, along with family member and recipient input, developed the Behavioral Health System Outcomes Framework and Dashboard as a means to focus departmental planning efforts by evaluating the effectiveness of the system using standardize measures, such as, family and recipient identified outcomes, the National Outcomes Measures (NOMs) and ADHS/DBHS performance metrics.  ADHS/DBHS foresees that the development of the Outcomes Framework will impact the future Consumer Survey results due to increased transparency, provider education and accountability in service provision. The development of the Outcomes Framework is in direct response to the identified need for transparency, accountability, improved family, recipient and provider education and tracking of outcomes as a means to drive both individual recipient and program outcomes. 2010 Consumer Survey Report September, 2011 Children’s Initiatives:  ADHS/DBHS continues to use the Children‟s System of Care Practice Review (CSOCPR) process to assess Child and Family Team (CFT) practice. These intensive case reviews include outcomes-focused questions to ensure CFTs are identifying and tracking outcomes as well as evaluating the effectiveness of provided services and improved functioning. The reviews are conducted at the individual provider level and are used to develop annual performance improvement plans. The plans are approved by DBHS and monitored by the RBHA with regular updates provided to DBHS.  The continued emphasis on in-home direct support services, as established by the Meet Me Where I Am (MMWIA) Campaign, has been a valuable resource to children and families with the most complex needs. These services put direct support workers in homes, schools and communities at the times and for as many hours as necessary to meet the family‟s needs, leading to improved functioning and better outcomes.  The annual Children‟s System of Care (CSOC) Strategic Plan (SP) provides a vehicle for the Division and RBHAs to address key goals around CFT Practice Model monitoring through the Practice Review process, and development of Generalist Direct Support services and High Needs Case Management services to work with children and families with the most complex needs.  The strategic activities prioritized in CSOC-SP continue to promote family driven care as well as youth and family involvement in local and statewide committees and system development activities. Focus is placed on specific populations such as children ages birth to 5, transition aged youth, and youth with substance use disorders in order to more clearly define their needs and help providers develop expertise in working with these youth. Specific Initiatives by Regional Behavioral Health Authorities: 18  Cenpatico has recognized the low scores in the Outcomes and Improved Functioning domains in both GSAs 2 and 4, and reported steps they are taking to address them. A Data Integrity Performance Improvement Plan was initiated to assure more timely collection of accurate information. Increased visibility of dashboards with functional outcome measures will help to highlight these efforts. A Spirit of Engagement Performance Improvement Project was also developed to implement a broader approach to improving care delivery, with a goal of strengthening client outcomes.  To address a statistically significant decrease in the youth survey Outcomes domain, CPSA-5 noted in their report of survey findings that outcomes will be a main focus of Comprehensive Service Providers‟ (CSPs) Performance Improvement Plans in the coming year. The potential changes to improve client outcomes are listed below. These initiatives are expected to also positively impact the adult survey Outcomes scores.  Outcome indicators will be added to forms that clients complete, opening the opportunity for consistent outcomes-oriented discussion; 2010 Consumer Survey Report September, 2011  Work groups will implement approaches to identify, communicate, and document outcomes with the child and family team;  MHSIP Outcomes language will be used in supervision practice;  Training by Recovery Support Specialist (RSS) staff for new and current employees will highlight the use of MHSIP outcomes language in discussing progress with clients. ADHS/DBHS will follow up with CPSA in response to their survey report and recommend periodic follow-up on their initiatives. Additionally, CPSA will be asked for an explanation of the low scores in the Improved Functioning domain, and to implement related initiatives. ADHS/DBHS will monitor their progress in the FY2012 Consumer Surveys. IX. Conclusion The statewide 2010 survey results indicate overall improvement across domains for both the Adult Consumer Survey and YSS-F when compared to 2009 surveys. This is the first year since survey inception that all domains have demonstrated improvement. Statistically significant increases were noted in the domains of Cultural Sensitivity (Youth) and Participation in Treatment Planning and Social Connectedness (Adult). As per the ADHS/DBHS Contract with the RBHAs, Consumer Satisfaction with Service Outcomes on the Annual Consumer Survey is a performance incentive; a score of 70% or greater on the Outcomes domain of the Consumer Survey makes up 25% of the incentive award. Adult and Youth survey results are evaluated independently, each contributing to half of the award. ADHS/DBHS believes that assisting members in the identification and tracking of treatment outcomes is imperative to improving behavioral health recipients‟ overall perception of the quality of services they receive. 19 2010 Consumer Survey Report September, 2011 ATTACHMENT-A 20 2010 Consumer Survey Report September, 2011 2010 Tribal Behavioral Health Survey Results Arizona Tribal BHAs (TBHAs) Gila River Indian Community, Pascua Yaqui Centered Spirit Program, Navajo Nation and White Mountain Apache Tribe participated in the 2010 Consumers Survey. The Tribal BHAs have shown an overall increase in participation from 236 participants in 2008 to 381 in 2010. All Tribal BHAs conduct a convenience sampling of their enrolled members. Overall, the number of surveys completed by tribal participants increased this year, particularly by Navajo and PYT adults. Number of Surveys Completed TBHA Gila River Navajo Nation Pascua Yaqui White Mountain Apache Statewide TBHA Adult Survey 71 YSS-F 35 Total 106 50 37 87 111 23 255 45 9 126 156 32 381 Methodology and Survey Administration A convenience sampling methodology was used (rather than a random sampling), therefore, all persons enrolled with the TBHAs were eligible for participation in the survey. Data Analysis TBHAs results were submitted directly to ADHS/DBHS and entered into the database; analyses were conducted by ADHS/DBHS staff. Summary of Results: The General Satisfaction, Service Quality and Appropriateness and Social Connectedness domains show some of the most positive responses for adults. Family of children receiving services reported the highest rates of satisfaction to services in most Tribal BHAs. The most positive scores for adults were expressed in the General Satisfaction and Service Access domains. Family of youth receiving services through the TBHAs reported relatively high rates of satisfaction in all domains except Outcomes and Improved Functioning. 21 2010 Consumer Survey Report September, 2011 Summary of 2010 TBHA Results for Adults General Satisfaction Service Access Participation in Treatment Planning Service Quality & Appropriateness Outcomes Improved Functioning Social Connectedness Gila River 93% 90% 92% 90% 77% 74% 88% Navajo Nation 89% 80% 79% 85% 73% 76% 75% Pascua Yaqui 95% 94% 89% 94% 87% 88% 91% White Mtn Apache 91% 95% 86% 91% 86% 83% 100% TBHA TBHA scores are not included in statewide numbers due to TRBHA convenience sampling methodology. Summary of 2010 TBHA Results for Youth General Satisfaction Service Access Participation in Treatment Planning Cultural Sensitivity Outcomes Improved Functioning Social Connectedness Gila River 91% 91% 97% 97% 57% 57% 94% Navajo Nation 86% 75% 78% 78% 75% 75% 72% Pascua Yaqui 84% 100% 84% 98% 58% 65% 93% White Mtn Apache 89% 100% 89% 100% 67% 67% 78% TBHA TBHA scores are not included in statewide numbers due to TRBHA convenience sampling methodology. 22 2010 Consumer Survey Report September, 2011 ATTACHMENT-B 23 2010 Consumer Survey Report September, 2011 2010 CONSUMER SURVEY PROTOCOL Carolyn Dempsey, PI Specialist, Quality Management Operations, Daniel Crough, Lead Programmer Arizona Department of Health Services Division of Behavioral Health Services 24 2010 Consumer Survey Report September, 2011 Executive Summary The 2010 consumer survey protocol reflects the changes implemented for the 2009 survey administration. Some revisions were incorporated last year in the sampling methodology and managerial responsibilities as compared to previous years and these changes proved to generally improve the survey‟s administration process. Two consumer surveys will be administered in 2010; one for adults and one for the parents/guardians of children receiving behavioral health services. The survey will be available to consumers in two languages: English and Spanish. Survey data will be entered into an MS Access database by the Regional Behavioral Health Authorities (RBHAs) and forwarded to the Arizona Department of Health Services/Division of Behavioral Health Services (ADHS/DBHS). ADHS/DBHS will data enter results for its tribal contractors (TRBHAs). Prior to implementation of the 2010 survey, ADHS/DBHS will provide each Geographic Service Area (GSA) with the CIS (Client Information System) Client IDs for 500 adult consumers and 400 to 500 youth consumers who the RBHAs/providers will encourage to participate in the survey. Each CIS Client ID will be linked to a Survey ID reference number to allow for tracking of which consumers completed a survey. The survey will be administered April through May 2010. As in previous years, the primary administration route will be to distribute the survey at the provider sites. As the consumer checks in for their appointment, the survey questionnaire will be handed to them. The consumer will be requested to complete the questionnaire prior to their appointment, but will be allowed to finalize the survey after the appointment if needed, or be provided with an addressed, stamped envelope to mail the survey in to the RBHA if they did not have time to complete it in the office. A drop box will be provided at each provider site for consumers to drop off their completed surveys. In cases where consumers who are selected for the sample have home appointments (as opposed to clinic appointments) during the survey timeframe, the questionnaire will be completed at home and mailed using the addressed, stamped envelope to be provided with the survey. If the consumer does not have a clinic or home appointment scheduled during the survey administration period or no-shows for a scheduled appointment, a non-clinical staff at the RBHA may conduct the survey by phone. Providers will never have access to completed surveys or individual survey results. ADHS has the statewide oversight responsibility for implementation and analysis of the survey data. The RBHAs will be responsible for ensuring that providers strictly adhere to the protocol. The providers are primarily responsible for the survey administration. Survey Instruments Two MHSIP survey instruments are the Adult Consumer Survey and the Youth Services Survey for Families (YSS-F). The adult survey will be administered to adult consumers of behavioral 25 2010 Consumer Survey Report September, 2011 health services and the YSS-F will be administered to parents/guardians of children receiving behavioral health services. The MHSIP Adult Consumer Survey measures seven domains: (1) Service Accessibility; (2) Service Quality or Appropriateness (which includes one item concerning cultural sensitivity); (3) Consumer Participation in Treatment Planning; (4) Outcomes; (5) General Satisfaction; (6) Improved Functioning; and (7) Social Connectedness. In addition, the questionnaire includes a module of questions to determine the impact of services received on the recipient‟s involvement with the criminal justice system. All questions are scored using a Likert Scale of 1 through 5 as follows: 1=Strongly Agree, 2=Agree, 3=Neutral, 4=Disagree, and 5=Strongly Disagree. A Not Applicable option is also available if the question does not apply. The MHSIP YSS-F focuses on the following seven domain areas: (1) Service Accessibility; (2) Participation in Treatment Planning; (3) Cultural Sensitivity; (4) Satisfaction with Services; (5) Outcomes; (6) Social Connectedness; and (7) Improved Functioning. Additional questions solicit information about the youth‟s criminal justice contact and school attendance. Questions are scored with a five-point Likert Scale where 1=Strongly Agree, 2=Agree, 3=Neutral, 4=Disagree, and 5=Strongly Disagree. A Not Applicable option is also available if the question does not apply. The RBHA and/or provider may pre-print the following information on the survey tool prior to distribution to client:  RBHA Name  Provider Name  Survey Id  Provider Facility ID  Behavioral Health Category  Entitlement Status (Title XIX or XXI)  Distribution Method (Home, Clinic, or Phone)  Client enrollment with DDD Both survey tools have four main sections: (1) demographic section, (2) MHSIP survey questions, (3) one state-added question pertaining to cultural sensitivity, and (4) open-ended qualitative section. The demographic section provides descriptive information about the consumer‟s age, gender, race, ethnicity, and relationship of the person completing the survey to the service recipient. The second section of the survey contains the MHSIP standardized questions. They serve as benchmark tools to compare consumer perception of behavioral health systems across the nation. The third section consists of one state-added question that asks for consumer input regarding the inclusion of cultural preferences in the consumer‟s treatment planning. The fourth and final section of the survey contains open-ended questions to solicit consumer comments. Two questions are asked of consumers – focusing on identifying what has been most helpful with their services, and what the consumer believes would improve services. The section 26 2010 Consumer Survey Report September, 2011 entitled Other Comments is intended to provide consumers with an additional area on the survey to provide open-ended feedback on any issue. Confidentiality The front page of the survey questionnaire addresses confidentiality of the responses. Survey results are aggregated and not presented at an individual consumer level. The providers will never have access to completed surveys or individual survey results. Thematic analysis is conducted on written comments. Non-mandatory (Voluntary) The survey questionnaire likewise informs the respondent that participation in the survey is voluntary; every individual can choose to participate or not. It also notifies the respondent that non-participation will not affect the services they are currently or will receive in the future. Participating in the survey is one way the consumer may help improve the system they use. Languages The survey will be available to consumers in English and Spanish languages. Each survey form will be printed in English on one side and Spanish on the reverse side. For consumers with limited English proficiency and speak a language other than Spanish, the RBHA and/or the provider administering the survey will extend their best effort to translate the survey in the consumer‟s preferred language by utilizing the Language Line or other translation/interpretation services officially utilized by the RBHA or their provider. The extent of assistance provided in language translation should not attempt to define what the question means. Sampling Design Sample Frame The sample frame refers to the population eligible to take the survey. This is the pool from which ADHS/DBHS will randomly select the sampled population. ADHS/DBHS will create an adult and a youth sample from the CIS database for each GSA. Clients 18 or older on December 1, 2009 are grouped as adults, and clients under 18 are grouped as youth. The adult clients may be enrolled in any program: General Mental Health, Substance Abuse, or Seriously Mentally Ill. The sample frame will be composed of: (a) Client must be enrolled in FY2010; (b) Client must be Title XIX or Title XXI eligible in FY2010; (c) Client must have received a mental health service other than inpatient, transportation, laboratory and/or radiology services, or crisis; (d) The service must have occurred in the previous 6 months. 27 2010 Consumer Survey Report September, 2011 Drawing of Sample ADHS/DBHS will create a random sample and provide each GSA with a list of 500 adult clients to be surveyed. For the youth sample, the CBHS-2 and CPSA-3 GSAs will receive a list of 400 clients; the other GSAs will receive a list of 500 clients. Administering the Survey ADHS/DBHS will provide the RBHAs with a list of randomly selected clients. The RBHA will duplicate the Survey ID number on the survey tool to be offered that specific consumer. To protect the confidentiality of the consumer, none of the parties privy to this information (DBHS, RBHA, Provider) will link the survey responses to consumers. ADHS/DBHS will provide the RBHAs with a database to store the survey results. The RBHA will disseminate the surveys; track which consumers have completed the survey; track why surveys were not completed; enter the survey data into the provided database; and submit to ADHS/DBHS: the completed paper surveys, the database containing all survey results, the database detailing which consumers completed a survey and for every consumer that did not complete a survey, the reason the survey was not completed. The RBHAs will enter data into the two ADHS/DBHS provided databases but the RBHAs will not alter the databases in any way. The provider agency is responsible for identifying the specific provider location or site from which the consumer is presently receiving services. Survey Methodology Distribution Method The primary distribution method is a non-clinical staff at the provider office (i.e., clinic) handing the survey questionnaire to the consumer. As the consumer checks in for their appointment, s/he will be provided with a copy of the survey questionnaire to complete. If the consumer agrees to participate, s/he will be requested to complete the survey prior to his/her appointment. If the consumer is unable to complete the questionnaire, s/he will be allowed to finish it on site after the appointment or be provided with an addressed, stamped envelope to mail the survey in if they did not have time to complete it in the office. A drop box will be provided on site for completed surveys. Additionally, a specific area at the provider office will be designated for completing the survey. If the individual randomly selected has a scheduled appointment at home during the survey window, the provider staff will bring the survey questionnaire at the appointment date. If the consumer agrees to participate, s/he will be advised to complete the survey after the staff leaves 28 2010 Consumer Survey Report September, 2011 and to mail the completed questionnaire to the RBHA using the pre-addressed, stamped envelope provided with the survey. If the individual does not have an appointment during the survey window, a non-clinical staff at the RBHA may conduct the survey over the phone. As an alternative, a non-clinical staff at the provider site may contact the consumer by phone to ask for their participation in the survey, offering a return envelope for the completed survey to be mailed. All return envelopes provided as a means for the consumer to submit their completed survey must be addressed to the RBHA. A check box in the questionnaire will be used to track the distribution method. The adult survey will be administered to the adult consumer. If the individual requests assistance, a guardian may complete the questionnaire on the consumer‟s behalf. The YSS-F will be administered to the parent/guardian of the child receiving services. If the parent or guardian is not at the appointment, then the survey will not be provided. List of Survey Clients ADHS/DBHS will provide each RBHA with a list of survey consumers to track those who complete a survey or the reason they did not complete a survey. The list of survey consumers will contain fields for:              RBHA ID Contractor ID Survey ID AHCCCS ID Last Name First Name Date of Birth Sex Survey Offered Date Survey Completed Date Reason Not Completed Provider ID Client contact phone number The RBHAs will submit their lists of survey clients to ADHS/DBHS bi-weekly throughout the survey administration period to keep ADHS/DBHS informed regarding the status of response rates and reasons for non-participation of sampled consumers. Due dates for submission of biweekly lists are included in the Task Timeline section on page 9 of this Protocol. Since this document contains protected health information, it will be treated as a confidential document. 29 2010 Consumer Survey Report September, 2011 ADHS/DBHS Post-administration Follow-up During the survey administration period, RBHAs will attempt to achieve participation of all consumers on its list of survey clients. Every effort will be made to separate service delivery from sample collection. At the end of the administration period, RBHAs will provide ADHS/DBHS with the completed list of survey consumers indicating which consumers completed the survey and which did not, with the reasons and a contact phone number for those who did not participate. At that point, the RBHA will delete the list of survey consumers. RBHAs will submit to ADHS/DBHS the completed paper surveys and a database containing survey results by July 30, 2010. Tribal RBHAs participate in the Annual Consumer Survey by way of a convenience sampling of their enrolled consumers. These entities will be provided 100 surveys for each population for distribution. Survey Administration Survey Timeframe The survey will be administered for a period of two months: April 1 through May 28, 2010. If participation rates are low, ADHS/DBHS may extend the survey period. Roles and Responsibilities ADHS/DBHS is responsible for the statewide oversight of the survey administration to ensure consistent implementation of the survey protocol. The protocol, client sample, survey instruments, and survey results database will be created by ADHS/DBHS. ADHS/DBHS will provide technical assistance throughout the survey process. Periodic monitoring, training, timelines, and use of checklist will be utilized to guide the T/RBHAs on critical points in the process. The T/RBHAs have the primary responsibility for ensuring that the protocol is precisely followed within their geographic regions. Direct oversight and assistance will be provided by the T/RBHAs to their providers. The T/RBHAs will ensure that the providers are appropriately trained and prepared to administer the survey. The RBHAs will enter data into the two ADHS/DBHS provided databases but the RBHAs will not alter the databases in any way including creating tables, changing the names of tables or fields, or creating a different value coding scheme. Each provider agency is primarily responsible for each of its sites in which the survey is to be administered. Each site will maintain all necessary materials for survey administration. At each site, a drop box and a designated area will be provided for consumers to complete the survey. Providers will also be responsible for the day-to-day operations – including having the survey 30 2010 Consumer Survey Report September, 2011 tools, materials for completing the survey (pens, pencils, clipboards), envelopes for return of the survey if needed, assigned resources for administration and collection of data for the survey. Non-Randomly Selected Consumers (Walk-in Requests) ADHS/DBHS does not require the RBHAs to extend survey participation efforts beyond the randomly selected populations but at the individual discretion of each RBHA/GSA, the RBHA may furnish its provider agencies with survey forms to give consumers who express a desire to participate in the survey but whose names do not appear on the lists of survey consumers (i.e., the consumer was not randomly selected). However, to maintain the scientific rigor of the protocol, the survey questionnaire completed by this group of respondents will be tracked separately by the RBHA. Names of individuals who belong to this group will not be added to the survey client list. This group will be tracked in some other ways as described in the succeeding paragraph. If the RBHA elects to collect responses from non-selected consumers, the same protocol will be followed for this group of respondents but the staff administering the survey will ensure that these “voluntary” surveys are kept separate from those on the lists of survey clients (randomly selected consumers). Several control measures will be used. First, consumers that are not randomly selected to participate in the survey (i.e., walk-in respondents) will be given a copy of the survey questionnaire with the pre-filled section of the questionnaire not completed. Second, the survey questionnaire that will be used for non-randomly selected consumers will be colorcoded. (Note: Surveys for the random sample will be printed on white paper.) RBHAs will enter the responses on the voluntary surveys into the same database as the results of the selected respondents, using the correct indicator, “C” for control (selected) respondents and “V” for voluntary (non-selected) respondents. Pre-Survey Activities Notification to Consumers about the Survey To encourage greater participation, efforts will be made to inform consumers in advance about the survey. T/RBHAs and providers will be encouraged to utilize all or a combination of any of the following media: flyers, posters, website announcements, or other promotional materials. A staff member or members at each provider site will be assigned to work on the survey. In addition to daily survey administrative duties, a component of the staff member‟s role will include assisting consumers with the survey if necessary. Assistance may include: reading the survey to individuals unable to read, explaining the Likert scale used for scoring answers, emphasizing confidentiality of the survey, or ensuring consumers that participation in the survey is voluntary. Staff will be allowed to provide administrative assistance to the consumer, or provide encouragement to participate in the survey process. However, staff will not be able to explain the meaning of particular questions or provide interpretations on what particular questions mean. 31 2010 Consumer Survey Report September, 2011 Data Management and Reporting Scoring Protocol For surveys to be included in the Arizona-wide/RBHA report, the client must have been selected in the sample population. Clients who are not part of the survey may ask to complete a survey; these surveys are labeled “volunteer” and are excluded from the statewide and RBHA results. Tribes select clients to complete the survey by convenience (not randomly selected from a sample frame). Tribal samples are excluded from the statewide totals. Responses to individual questions are considered invalid when the response cannot be determined. This occurs most often when no response is marked, but can also occur if more than one response is marked. Some questions depend on the answer to other questions, e.g., “If you have received care from this provider for more than 6 months then answer the following questions.” These questions will have a different total number of responses. The scoring protocol recommended by MHSIP will be utilized for evaluating the domain areas within the survey, as follows: 1. Recode ratings of „not applicable‟ as missing values. 2. For each survey, exclude domains with one-third or more of the domain questions missing. 3. Calculate the mean of the items for each respondent. 4. Calculate the percent of scores that are less than 2.5 Technical assistance ADHS will provide technical assistance to the RBHAs as needed. Response Rate Calculation The rate will be calculated for each population for each GSA using the formula: Response rate = A / B Where: A= Total number of surveys returned B= Total number of clients in the sample that were contacted and asked to participate 32 2010 Consumer Survey Report September, 2011 Weighting Methodology The statewide data will be weighted by GSA client population to compensate for the stratified sample collection. Weights will be applied to the survey data prior to any statewide data analysis. Dissemination of Findings The following methods have been identified to circulate findings: Reporting of survey results in management meetings – Executive Management, Quality Management, Human Rights Committees, Behavioral Health Planning Council, other consumer advocate groups, and other interest groups that may be identified  Dissemination of information by providers to their local communities  Having copies of the survey available at the provider sites  Publishing results of survey on ADHS, RBHA and provider websites  Timeline for tasks to be completed by RBHA and ADHS/DBHS:       March 1, 2010: ADHS/DBHS provides each RBHA with: Lists of consumers in survey samples Survey tools, Adult and YSS-F – Survey tools in Adobe format to be copied by RBHA (white for selected respondents; light green for voluntary respondents). Header portion of survey tools will be editable for RBHA entry of information called for. Database – for RBHA data entry of survey results. March: RBHAs identify appropriate provider for consumers on lists of survey consumers, copy ADHS/DBHS-provided survey tools, complete top portions specific to each consumer and forward survey tools to providers. April 1 – May 28, 2010: Administer survey April 19, April 30, May 14, June 11, 2010: RBHA submits to ADHS/DBHS: Lists of survey clients – the lists of survey clients with current status regarding which consumers have and have not completed a survey. July 30, 2010: RBHA submits to ADHS/DBHS: Database containing survey results. Completed surveys – this includes surveys completed by selected respondents as well as voluntary respondents. A copy of the surveys is to be retained by the RBHA. December 3, 2010: RBHA submits a written report of survey findings to ADHS/DBHS. This report will provide analysis of 2010 consumer survey results and performance improvement activities planned or implemented to address areas in need of improvement. List of Attachments to 2010 Statewide Consumer Survey Protocol 1. 33 2010 Consumer Survey Tool: Question & Domain Key 2010 Consumer Survey Report 2. 2010 Consumer Survey: Domains & Corresponding Questions 3. Year 2010 Adult Consumer Survey 4. Year 2010 Youth Services Survey for Families 34 September, 2011 2010 Consumer Survey Report September, 2011 ATTACHMENT-C (See attached Excel Raw Data File) 35 2010 Consumer Survey Report September, 2011 ATTACHMENT-D 36 2010 Consumer Survey Report September, 2011 2010 Adult Demographics 2010 Adult Consumer Survey, Percent of Positive Response by Domain and Subgroup Subgroup General Satisfaction Service Access Service Quality and Appropriateness Participation in Treatment Planning Outcomes Improved Functioning Social Connectedness N % N % N % N % N % N % N % 534 799 86 87 525 742 82 80 564 837 89 91 553 806 92 91 489 657 78 72 477 612 76 68 500 729 79 78 73 222 433 577 30 2 85 78 85 92 83 100 74 220 436 510 29 2 87 78 81 81 83 100 75 249 463 575 36 2 90 88 88 92 95 100 67 249 462 560 22 2 89 93 90 93 92 100 64 203 403 453 27 2 75 72 78 74 79 100 58 194 373 439 24 2 74 69 73 71 71 100 70 230 413 496 30 2 85 80 78 79 86 100 940 94 16 37 86 87 94 90 904 87 15 37 82 78 88 84 998 98 18 41 90 90 100 93 974 88 16 34 92 87 94 87 819 77 16 33 75 69 89 73 773 74 16 34 71 69 89 77 864 86 14 44 77 77 100 98 2 100 2 100 2 100 2 100 2 100 1 50 2 100 39 77 35 67 35 73 34 79 26 52 24 49 36 68 337 850 89 85 315 816 84 80 345 907 92 89 351 877 94 91 276 757 74 75 259 712 70 71 312 787 83 76 122 101 230 246 612 90 81 83 85 89 118 97 221 243 561 86 80 80 82 79 126 107 251 271 619 93 88 92 92 87 121 107 243 268 599 93 94 92 92 91 98 85 211 211 517 74 73 76 73 75 94 92 204 202 475 70 79 74 71 69 119 97 225 224 544 84 80 81 77 77 447 623 86 88 435 593 82 84 469 632 88 91 470 629 91 92 384 526 73 76 364 501 69 73 416 555 78 79 Gender Male Female Age Group 18-21 22-30 31-45 46-65 66-75 75+ Race White only African American only Asian only Am Indian/Al Native only Nat Hawaiian/Pacific Islander only Multiple Race Ethnicity Hispanic or Latino Not Hispanic or Latino Length of Services 0-6 months 7-11 months 1-2 years 3-5 years 5 years + Program SMI Non-SMI 37 2010 Consumer Survey Report September, 2011 ATTACHMENT-E 38 2010 Consumer Survey Report September, 2011 2010 Adult Line Item 2010 Adult Consumer Survey, Statewide Percent of Positive Response by Line Item (Line specific numbers are based on actual valid survey returns. Domain percentages are based on weighted scores.) Survey Item General Satisfaction: 1. I like the services that I received here. 2. If I had other choices, I would still get services from this agency. 3. I would recommend this agency to a friend or family member. Service Access: 4. The location of services was convenient (parking, public transportation, distance, etc.) 5 Staff were willing to see me as often as I felt it was necessary.. 6. Staff returned my call in 24 hours. 7. Services were available at times that were good for me. 8. I was able to get all the services I thought I needed. 9. I was able to see a psychiatrist when I wanted to. Participation in Treatment Planning 11. I felt comfortable asking questions about my treatment and medication. 17. I, not staff, decided my treatment goals. Service Quality and Appropriateness 10. Staff here believe that I can grow, change and recover. 12. I feel free to complain. 13. I was given information about my rights. 14. Staff encouraged me to take responsibility for how I live my life. 15. Staff helped me to understand what side effects to watch out for. 16 Staff respected my wishes about who is and who is not to be given information about my treatment.. 18. Staff were sensitive to my cultural background (race, religion, language, etc.) 19. Staff helped me obtain the information I needed so that I could take charge of managing my illness. 20. I was encouraged to use consumer-run programs (support groups, drop-in centers, crisis phone line, etc.) Outcomes 21. I deal more effectively with daily problems. 22. I am better able to control my life. 23. I am better able to deal with crisis. 24 I am getting along better with my family. 25. I do better in social situations. 26. I do better in school and/or work 27. My housing situation has improved 28. My symptoms are not bothering me as much. 39 Number Percent 1345 1405 1313 1373 1280 86 87 82 85 81 1286 1322 1159 1351 1303 1113 1372 1425 1276 1410 1356 1359 1445 1363 1279 80 82 75 84 81 72 92 90 82 90 85 85 90 86 81 1465 92 1287 85 1309 84 1249 1161 1213 1205 1144 1117 1081 817 952 1047 82 75 76 76 73 73 70 63 65 66 2010 Consumer Survey Report Improved Functioning 28. My symptoms are not bothering me as much. 29. I do things that are more meaningful to me. 30. I am better able to take care of my needs. 31. I am better able to handle things when they go wrong. 32 I am better able to do things that I want to do. Social Connectedness 33. I am happy with the friendships I have. 34. I have people with whom I can do enjoyable things. 35. I feel I belong in my community. 36. In a crisis, I would have the support I need from family or friends. 40 September, 2011 1099 1047 1112 1164 1133 1097 1247 1194 1247 1027 1241 71 66 72 74 73 71 79 75 79 65 79 2010 Consumer Survey Report September, 2011 ATTACHMENT-F 41 2010 Consumer Survey Report September, 2011 2010 YSS-F Demographics 2010 YSS-F, Percent of Positive Response by Domain and Subgroup General Satisfaction Subgroup Service Access Cultural Competency Participation in Treatment Planning Outcomes Improved Functioning Social Connectedness N % N % N % N % N % N % N % 756 424 85 85 710 411 83 83 835 467 96 96 775 443 92 90 619 342 70 68 628 345 71 70 783 415 88 85 74 687 436 94 87 80 69 639 435 86 84 81 76 728 512 99 96 96 73 697 470 97 93 89 45 550 380 63 71 67 44 558 386 63 72 69 59 681 478 84 89 86 732 89 3 38 84 86 75 81 682 95 4 44 81 86 100 86 807 108 4 51 97 97 100 100 792 90 4 41 93 88 100 82 623 62 3 32 71 58 75 63 633 60 4 32 73 58 100 64 745 94 3 39 86 90 75 87 4 52 50 77 4 64 50 93 7 60 100 94 3 59 43 92 3 39 38 57 3 43 38 63 7 62 100 89 532 567 88 82 479 567 83 83 571 640 97 96 534 604 94 90 441 456 73 66 441 472 74 69 549 571 93 84 203 196 371 213 156 87 86 87 81 82 192 183 354 197 152 82 84 86 78 82 223 210 399 242 173 97 98 98 95 93 196 204 380 237 169 88 96 93 94 90 139 168 315 184 128 60 74 73 72 66 137 166 321 187 132 61 74 75 73 70 199 196 382 214 163 87 88 90 85 84 826 287 86 82 779 290 84 82 897 324 97 94 862 292 94 86 654 257 68 72 665 257 70 73 843 301 90 85 Gender Male Female Age Group 0-4 5-12 13-17 Race White only African American only Asian only Am Indian/Al Native only Nat Hawaiian/Pacific Islander only Multiple Race Ethnicity Hispanic or Latino Not Hispanic or Latino Length of Services 0-6 months 7-11 months 1-2 years 3-5 years 5 years + CFT Yes No 42 2010 Consumer Survey Report September, 2011 ATTACHMENT-G 43 2010 Consumer Survey Report September, 2011 2010 YSS-F Line Item 2010 YSSF, Statewide Percent of Positive Response by Line Item (Line specific numbers are based on actual valid survey returns. Domain percentages are based on weighted scores.) Survey Item General Satisfaction: 1. Overall, I am satisfied with the services my child received. 4. The people helping my child stuck with us no matter what. 5. I felt my child had someone to talk to when he/she was troubled. 7. The services my child and/or family received were right for us. 10. My family got the help we wanted for my child. 11. My family got as much help as we needed for my child. Service Access 8. The location of services was convenient for us. 9. Services were available at times that were convenient for us. Participation in Treatment Planning 2. I helped to choose my child‟s services. 3. I helped to choose my child‟s treatment goals. 6. I participated in my child‟s treatment. Cultural Sensitivity 12 Staff treated me with respect. 13. Staff respected my family‟s religious/spiritual beliefs. 14. Staff spoke with me in a way that I understood. 15. Staff were sensitive to my cultural/ethnic background. Outcomes 16 My child is better at handling daily life. 17. My child gets along better with family members. 18. My child gets along better with friends and other people. 19. My child is doing better in school and/or work. 20. My child is better able to cope when things go wrong. 21. I am satisfied with our family life right now. 22. My child is better able to do things he or she wants to do. Improved Functioning 16 My child is better at handling daily life. 17. My child gets along better with family members. 18. My child gets along better with friends and other people. 19. My child is doing better in school and/or work. 20. My child is better able to cope when things go wrong. 22. My child is better able to do things he or she wants to do. Social Connectedness 23. I know people who will listen and understand me when I need to talk. 24. I have people that I am comfortable talking with about my child‟s problems. 25. In a crisis, I would have the support I need from family or friends. 26. I have people with whom I can do enjoyable things. 44 Number 1197 1243 1137 1110 1210 1136 1080 1143 1161 1128 1239 1185 1310 1325 1316 1345 1224 1364 1164 975 1020 992 1000 987 893 894 983 988 1020 992 1000 987 893 983 1219 1191 1239 1158 1179 Percent 85 86 80 80 85 80 76 83 82 80 92 84 93 94 96 94 91 96 90 69 72 70 71 71 64 64 70 71 72 70 71 71 64 70 88 86 88 83 85 2010 Consumer Survey Report September, 2011 ATTACHMENT-H 45 2010 Consumer Survey Report September, 2011 List of Acronyms 2010 Consumer Survey Report Acronyms ADHS/DBHS ASOC CBHS CFT CPSA CSOC CSOCPR CSR GMH GSA MHSIP MMWIA NARBHA NASMHPD NOMs NRI QM RSS SA SAMSHA SMI T/RBHA WSDSG YSS-F 46 Description Arizona Department of Health Services/Division of Behavioral Health Services Adult System of Care Cenpatico Behavioral Health Services Child and Family Team Community Partnership of Southern Arizona Children‟s System of Care Child System of Care Practice Review Consumer Survey Report General Mental Health Geographical Service Area Mental Health Statistics Improvement Program Meet Me Where I Am Northern Arizona Regional Behavioral Health Authority National Association of State Mental Health Program Directors National Outcomes Measures National Research Institute Quality Management Recovery Support Specialist Substance Abuse Substance Abuse and Mental Health Services Administration Serious Mental Illness Tribal/Regional Behavioral Health Authority Western States Decision Support Group Youth Services Survey for Families 2010 Consumer Survey Report September, 2011 APPENDIX-1 (See attached) Year 2010 Adult Consumer Survey (English and Spanish) 47 2010 Consumer Survey Report September, 2011 APPENDIX-2 (See attached) Year 2010 Youth Consumer Survey for Families (English and Spanish) 48