Report 2012-122 All Recommendation Responses

Report 2012-122: Mental Health Services Act: The State's Oversight Has Provided Little Assurance of the Act's Effectiveness, and Some Counties Can Improve Measurement of Their Program Performance (Release Date: August 2013)

Recommendation #1 To: Health Care Services, Department of

To ensure that it monitors counties to the fullest extent as the MHSA specifies and that it implements best practices, Health Care Services should draft and enter into a performance contract with each county that contains assurances for effective oversight and furthers the intent of the MHSA, including demonstration that each of the county's MHSA programs are meeting the MHSA's intent.

Annual Follow-Up Agency Response From October 2015

DHCS has incorporated provisions in the County Performance Contract that specify best practices for monitoring and oversight and provide guidance to counties to effectively demonstrate the implementation of their MHSA funded programs. DHCS outlines Department Compliance Investigations and County Breach, Plan of Correction and Withholding of State Mental Health Funds processes in each Performance Contract. Of the contracts submitted, 47 have been fully executed and six are in final processing.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

As of June 30, 2014, 36 counties submitted their Board of Supervisor's approved FY 2013-14 Performance Contracts to DHCS. Of the contracts submitted, 22 are fully executed and 14 are in final processing.

DHCS has continued to work with the County Behavioral Health Director's Association and the California State Association of Counties to address concerns regarding the applicability of certain requirements contained within the contract. Having completed these contract discussions, the FY 2014-15 Performance Contracts were released to the counties on July 28, 2014 with a due date for submission to DHCS within 45 days (by September 12, 2014).

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

Performance Contracts were forwarded to counties on September 27, 2013, and as of January 21, 2014, 24 counties have submitted their Board of Supervisors approved Performance Contracts to DHCS. Of the 24 contracts submitted, 18 are fully executed and 6 are in final processing. Three contracts were calendared at local Boards of Supervisor meetings during January 2014.

The California Mental Health Directors Association (CMHDA) has informed DHCS that many counties are stalled on processing the Performance Contracts because the California State Association of Counties (CSAC) counsel has concerns about the applicability of certain requirements contained within a contract exhibit. DHCS is researching this issue and is scheduled to meet on February 10, 2014 with CMHDA and their CSAC Counsel to address their concern.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The Performance Contract contains language that allows the Department of Health Care Services (DHCS) to monitor a county's performance according to the provisions of the Mental Health Services Act (MHSA) and related regulations. The Performance Contract provides that a county may be required to develop a plan of correction with specific timelines regarding any findings. DHCS may, after a hearing held with no less than 20 days notice to the county mental health director, withhold payments from the MHS Fund until the County is in compliance. The Performance Contract requires MHSA data reporting of full service partnerships, the achievement of performance outcomes, and of revenues and expenditures.

The Performance Contract will be sent to the counties for signature approval in September 2013. The time required to fully execute this contract will depend on how quickly each county is able to gain Board of Supervisor approval. In the next report, this recommendation's implementation status will be updated with the number of counties that have fully executed their Performance Contract.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #2 To: Health Care Services, Department of

To ensure that it monitors counties to the fullest extent as the MHSA specifies and that it implements best practices, Health Care Services should conduct comprehensive on-site reviews of county MHSA programs, including verifying county compliance with MHSA requirements.

Annual Follow-Up Agency Response From August 2019

Implementation of the MHSA program reviews commenced January 2019 for a total of 17 counties scheduled for 2019. As of July 2019, 13 county program reviews have been completed and the final four counties will be completed by September 23, 2019. The MHSA program review schedule has been finalized for the 2020 year with 19 counties scheduled for review. A total of 59 MHSA program reviews will be completed by 2021 and reviews will be conducted on a triennial schedule.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Health Care Services started MHSA program reviews in January 2019 including verifying county compliance with MHSA requirements.

Annual Follow-Up Agency Response From November 2018

Department of Health Care Services (DHCS) developed an evaluation protocol/review program to perform onsite program reviews of county Mental Health Services Act programs and piloted the protocol in February 2018. DHCS revised the protocol and provided it to stakeholders for review and input in May 2018.

DHCS selected four county program reviews for pilot testing during the months of July, August, September, and October 2018. Full implementation of the protocol review program for all 58 county site visits will commence January 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From August 2017

DHCS' Triennial Medi-Cal review includes MHSA questions, and compliance data from FY 2014-15 through FY 2016-17. There are 16 total questions in the review protocol. For 8 of the questions, 100% of the counties are in compliance with all requirements. For the other 8 protocol questions, there are 1-4 counties out of compliance. DHCS continues to conduct fiscal audits of local MHSA funds and, if needed, targeted audits of state administrative funds. To date, there are five counties that have been or are being audited and approximately two more scheduled for the next production year. There remains to be a high amount of carryover work from FY 16-17 to FY 17-18. DHCS is recruiting staff to perform onsite program reviews of county performance contracts. Once staff are hired and the review process and protocol is finalized, DHCS will begin performing onsite program reviews of county MHSA programs. DHCS is planning to combine these onsite reviews with the triennial Medi-Cal reviews, which are expected to begin in the Fall of 2017.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

DHCS' Triennial Medi-Cal review includes MHSA questions. DHCS continues to conduct fiscal audits of local MHSA funds and, if needed, targeted audits of state administrative funds. In July 2016, AB 1618 was signed into law requiring DHCS to conduct program reviews of county performance contracts, with each county being reviewed on a triennial basis. DHCS expects to complete the process and protocol for program reviews and begin county reviews by June 2017. Once program reviews are implemented, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

DHCS has incorporated MHSA questions in the department's Triennial Medi-Cal review. Each county is reviewed at least once every three years. The Department has begun to implement fiscal audits of county mental health department's MHSA expenditures. The department has not implemented comprehensive program reviews as it is continuing to work on obtaining additional resources to implement those reviews. Once comprehensive program reviews are implemented, this recommendation will be fully implemented. In the meantime, DHCS is continuing to respond to concerns and questions that are raised through our Issue Resolution Process. (See Rec. 2 Attachment)

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

Program reviews for FY 2013-14 were completed in June 2014. During FY 2013-14, Sutter/Yuba, Kings, Orange, Merced, Alameda, Stanislaus, Santa Barbara, San Mateo, Ventura, Humboldt, Contra Costa, Sonoma, San Francisco, Tuolumne, San Benito, Marin, Nevada, and Shasta counties were reviewed. Final reports have been issued to six counties; of these counties five are in compliance with the MHSA Section of the Medi-Cal Specialty Mental Health Review Protocol. Final reports for the twelve remaining counties are expected to be released by early October 2014.

DHCS is developing the FY 2014-15 Medi-Cal Specialty Mental Health Services review protocol. MHSA questions have been revised and new questions are being added to Section J: Mental Health Services Act. The final review protocol and schedule will be released during August 2014.

DHCS is currently developing a review tool which will be utilized to conduct onsite program reviews. Approximately 10 county site reviews will be conducted per year.

DHCS Audits and Investigations (A&I) began its MHSA audit of San Diego County on February 18, 2014. A&I expects to complete the audit for Fiscal Year's (FY) 07-08 and FY 08-09 by September 30, 2014. A final audit report is expected to be issued by October 30, 2014. Based on the San Diego audit, A&I will further refine the audit protocol.

DHCS will also conduct an MHSA audit of Solano County for FY 08-09 and FY 09-10. The entrance conference is scheduled on September 10, 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

This recommendation is partially implemented because Health Care Services' Medi-Cal Specialty Mental Health Review Protocol includes a limited number of questions targeted to MHSA and thus, is not a comprehensive on-site review. Moreover, Health Care Services is still developing a tool to conduct onsite program reviews.

6-Month Agency Response

DHCS performs a Medi-Cal Specialty Mental Health Services system review of each County Mental Health Plan on a triennial basis. MHSA reporting requirement questions were added to the FY 2013-14 Annual Review Protocol for Consolidated Specialty Mental Health Services and Other Funded Services. The FY 2013-14 reviews will begin in October 2013. Eighteen counties will be reviewed during the FY 2013-14. .

Any county out of compliance with any items will be required to submit a plan of correction, which DHCS will monitor for implementation.

DHCS performs fiscal audits (FA) of county cost reports annually. Currently, DHCS Audits and Investigations (A&I) is reviewing FY 2008-09 fiscal records and reconciling to the FY 2008-09 county cost reports. MHSA funds and the MHSA Annual Review and Expenditure Report will be included in the scope of the cost report audits. The goal of the FA is to help ensure funds are being spent according to the statutory requirements. These audits may include, among other protocols, tracing MHSA funds from the State Controller to the counties and providers and ensuring MHSA funds earn interest, have appropriate prudent reserves, and have not supplanted other funds.

Counties are selected for audits based on 1) a risk assessment performed by DHCS A&I or 2) a referral from DHCS MHSA programs. Currently, DHCS is finalizing the audit and appeals process and drafting corresponding regulations. Once finalized, DHCS will share the audit and appeals process with the Mental Health Services Oversight and Accountability Commission (Accountability Commission) and California Mental Health Directors Association (CMHDA). The three audit positions received as part of Assembly Bill 1467, Statutes of 2012, have been filled. The MHSA fiscal audits plans to commence January 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

Program Reviews

DHCS performs a Medi-Cal Specialty Mental Health Services system review of each County Mental Health Plan on a triennial basis. MHSA reporting requirement questions were added to the FY 2013-14 Annual Review Protocol for Consolidated Specialty Mental Health Services and Other Funded Services. The FY 2013-14 reviews will begin in October 2013. Eighteen counties will be reviewed during the FY 13 - 14. .

Any county out of compliance with any items will be required to submit a plan of correction, which DHCS will monitor for implementation.

DHCS performs fiscal audits (FA) of county cost reports annually . Currently, DHCS Audits and Investigations (A&I) is reviewing FY 2008-09 fiscal records and reconciling to the FY 2008-09 county cost reports. MHSA funds and the MHSA Annual Review and Expenditure Report will be included in the scope of the cost report audits. The goal of the FAis to help ensure funds are being spent according to the statutory requirements. These audits may include, among other protocols, tracing MHSA funds from the State Controller to the counties and providers and ensuring MHSA funds earn interest, have appropriate prudent reserves, and have not supplanted other funds.

Counties are selected for audits based on 1) a risk assessment performed by DHCS A&I or 2) a referral from DHCS MHSA programs. Currently, DHCS is finalizing the audit and appeals process and drafting corresponding regulations. Once finalized, DHCS will share the audit and appeals process with the Mental Health Services Oversight and Accountability Commission (Accountability Commission) and California Mental Health Director's Association (CMHDA). The three audit positions received as part of Assembly Bill 1467, Statutes of 2012, have been filled. The MHSA fiscal audits plans to commence January 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #3 To: Health Care Services, Department of

To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should coordinate with the Accountability Commission and issue guidance or regulations, as appropriate, for Facilities programs and for other MHSA requirements, such as a prudent reserve.

Annual Follow-Up Agency Response From October 2019

DHCS submitted proposed fiscal regulations to the OAL on March 8, 2019. The Notice of Proposed Rulemaking was posted on March 22, 2019. DHCS held the public hearing on May 7, 2019, and closed the 45-day comment period on May 17, 2019. DHCS has one-year from the date when the Notice of Proposed Rulemaking was posted to promulgate the fiscal regulations.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From August 2019

DHCS submitted proposed fiscal regulations to the Office of Administrative Law on March 8, 2019. The Notice of Proposed Rulemaking was posted on March 22, 2019. DHCS held the public hearing on May 7, 2019 and closed the 45-day comment period on May 17, 2019. DHCS has one-year from the date when the Notice of Proposed Rulemaking was posted to promulgate the fiscal regulations.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Health Care Services has submitted proposed regulations to the Office of Administrative Law and is engaged in the regulatory process.

Annual Follow-Up Agency Response From November 2018

Department of Health Care Services (DHCS) continues to update the regulations to reflect changes to reversion statutes implemented by the passage of Chapter 38, Statutes of 2017 (AB 114). DHCS is finalizing the draft fiscal regulation text and the initial statement of reasons. By January 2019, DHCS intends to submit the public notice that announces these proposed fiscal regulations and initiate the 45-day public comment period to the Office of Administrative Law for publication in the California Regulatory Notice Register. DHCS expects to fully implement this recommendation by Summer 2019, when regulations are fully promulgated.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2017

DHCS is developing MHSA fiscal regulations and the initial statement of reasons. The regulations will focus on three main areas: maintenance of records; prudent reserve; and reversion. DHCS convened a workgroup with representatives from the MHSOAC and the County Behavioral Health Directors Association to discuss the issues and provide input. DHCS expects the regulations to be available for public comment by December 2017 and promulgated during Summer 2018. Once regulations have been promulgated, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

DHCS submitted draft regulations for the Workforce Education and Training component to California Health and Human Services Agency for review. Upon review by Agency, and with input from the Office of Statewide Planning and Development, it was determined the proposed regulation changes were no longer necessary. DHCS continues to draft regulations for fiscal policies. A workgroup comprised of DHCS staff, MHSOAC staff, and counties has been developed to gather input for MHSA fiscal policies and regulations.

DHCS expects to complete draft regulations by December 2016, and fully promulgate the regulations by mid-2018. Once these regulations have been promulgated, this recommendation will be fully implemented

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

DHCS is currently working with the Office of Statewide Health Planning and Development to promulgate regulations on the Workforce Education and Training component. DHCS is also working on drafting regulations to provide more guidance to counties regarding other MHSA requirements, such as the prudent reserve. Once these regulations have been promulgated, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

DHCS continues to finalize the MHSA audit and appeals regulations, as well as draft the Initial Statement of Reasons. Draft regulations were provided to the County Behavioral Health Director's Association and the Mental Health Services Oversight and Accountability Commission for review and comment and DHCS is continuing to make revisions. These regulations will provide increased clarity to the audit and appeals process at the county level. DHCS is also working internally with the Office of Administration Hearings and Appeals to update the appeals regulations. The department expects to complete this regulation process by December 2015.

After drafting the audit and appeals regulations, DHCS will begin drafting fiscal regulations, CSS regulations, CFTN regulations, and, finally, WET regulations. DHCS anticipates completing all regulations by March 2018.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

DHCS completed the initial draft of MHSA audit and appeals regulations. The draft regulations will be provided to the California Mental Health Directors Association (CMHDA) by late February 2014 for review and comment. DHCS continues to work with the Mental Health Services Oversight and Accountability Commission to develop and update Community Services and Supports (CSS) regulations. When the CSS regulations are completed, fiscal regulations, Capital Facilities and Technological Needs (CF/TN), and Workforce Education and Training (WET) components will follow.

Draft regulations will be made available for public comment in the spring of 2014. We anticipate that final regulations will be adopted during the fall of 2014.

Per AB 82 (Chapter 23, Statutes of 2013) the Mental Health Services Oversight and Accountability Commission (Accountability Commission) is responsible for completing Prevention and Early Intervention (PEI) and Innovation (INN) regulations.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS is meeting with the Accountability Commission bi-weekly to review current MHSA regulations and to develop additional regulations. DHCS is currently reviewing regulations for Community Services and Supports (CSS), Capital Facilities/Technological Needs (CF/TN), fiscal policies including prudent reserve and reversion, and Workforce Education and Training (WET). DHCS is also developing program oversight and fiscal audit regulations.

Draft regulations will be made available for public comment in the spring of 2014. We anticipate that final regulations will be adopted during the fall of 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #4 To: Health Care Services, Department of

To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should commence this regulatory process no later than January 2014.

6-Month Agency Response

DHCS continues to work with the Accountability Commission to review the current MHSA regulations and to develop additional regulations.

DHCS completed the initial draft of MHSA audit and appeals regulations. The draft regulations will be provided to the California Mental Health Directors Association (CMHDA) by late February 2014 for review and comment. DHCS anticipates making these regulations available for public comment in the spring of 2014. We anticipate that final regulations will be adopted beginning in the fall of 2014.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DHCS began working in collaboration with the Accountability Commission in July 2013 to review the current MHSA regulations and to develop additional regulations. Draft regulations will be made available for public comment in the spring of 2014. We anticipate that final regulations will be adopted during the fall of 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #5 To: Health Care Services, Department of

To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should collaborate with the Accountability Commission to develop and issue guidance or regulations, as appropriate, to counties on how to effectively evaluate and report on the performance of their MHSA programs.

Annual Follow-Up Agency Response From October 2019

DHCS entered into a vendor contract to prepare the guidance recommended by the CSA. The contractor provided DHCS with recommendations on uniform procedures, metrics, and templates that counties could use for assessing and reporting the performance of MHSA programs. DHCS expects to release guidance to counties by December 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From August 2019

DHCS entered into a vendor contract to prepare the guidance recommended by the CSA. The contractor provided DHCS with recommendations on uniform procedures, metrics, and templates that counties could use for assessing and reporting the performance of MHSA programs. DHCS expects to release guidance to counties by December 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Health Care Services has received information from a vendor that could form the basis of guidance to counties for assessing and reporting the performance of MHSA programs; it expects to release that guidance by the end of 2019.

Annual Follow-Up Agency Response From November 2018

DHCS entered into a vendor contract to improve data quality and completeness in the CSI and the MHSA DCR. The vendor created reports of findings (e.g., data entry errors, anomalies, missing data) and delineated the need for improved infrastructure and maintenance of data quality. The vendor has also provided customized technical assistance (e.g., weekly webinars, monthly site visits) for counties and Electronic Health Record (EHR) vendors to improve data quality. For sustainability of improvement efforts, DHCS extended the vendor's contract to FY 2018-19 and is negotiating a contract for FY 2019-20. DHCS plans to transition vendor-provided technical assistance and system training to DHCS staff within the contract timeframe for continued county support. During FY 2019-20, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs. Full implementation is expected by June 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2017

DHCS has contracted with a vendor to perform data clean up on the data included in the DHCS Client Services Information System and MHSA Data Collection and Reporting System. The purpose of this project is to provide DHCS, the MHSOAC, and the counties with accurate and complete data that may be used to track performance outcomes for Community Services and Supports, Full Service Partnership clients. The project will be completed over the next 12-18 months, with DHCS maintaining data quality thereafter. Counties have already received their comprehensive data error reports, and technical assistance calls to support the interpretation and use of these reports are currently underway. After the data clean up project is complete, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

As a first step toward evaluating and reporting on MHSA-funded programs, DHCS determined that it is necessary to work with counties to improve the quality of the Full Service Partnership data reported to the DHCS MHSA Data Collection and Reporting System. DHCS sought, and was provided with, a re-appropriation of MHSA funds to strengthen data quality, which will be the focus of our efforts in FY 2016-17. Thereafter, DHCS will work with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

DHCS is planning to start discussions with the MHSOAC to determine the kind of guidance that needs to be produced to assist counties with effectively evaluating and reporting on the performance of their MHSA-funded programs.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

DHCS continues to support the Mental Health Services Oversight and Accountability Commission's (MHSOAC) Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, data monitoring and reporting for compliance, and ongoing data system planning and coordination meetings.

As part of the review and development of MHSA regulations, DHCS continues to consult with the Accountability Commission to determine any regulatory changes needed to ensure data are collected consistently and in support of the Evaluation Master Plan.

If needed, draft regulations will be developed in consultation with the Accountability Commission and made available for public comment in the winter of 2015, with final regulations adopted during the fall of 2016.

DHCS also continues the development of the Performance Outcomes System Plan for Medi-Cal Specialty Mental Health Services for Children and Youth (POS). In September of 2013, DHCS staff began reviewing the status of county data submission to the Client and Services Information (CSI) System. As of June 30, 2014, 79% of counties had submitted CSI data within the 60-day submission window. Data review and cleanup, as well as timely submission of data, is critical to the quality of data for the POS and other mental health services evaluation efforts. Beginning in October 2014, DHCS will provide quality improvement reports to the counties in order to prepare for the initial outcomes reports for specialty mental health, which are scheduled to be released in December 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

DHCS continues to support the Accountability Commissions Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, and data monitoring and reporting compliance.

DHCS continues its ongoing effort to identify outcomes/measures for children and youth as part of the Performance Outcomes System Plan for Medi-Cal Specialty Mental Health Services for Children and Youth (POS). The POS system plan was submitted to the Legislature on November 1, 2013 and the system implementation plan was submitted on January 1, 2014. Since some of the evaluation efforts in the POS and the Evaluation Master Plan are similar, DHCS and the Accountability Commission continue to collaborate to develop and issue guidance to counties on how to effectively evaluate and report on program performance.

DHCS continues to work with counties on their submission of mental health data. As part of the MHSA training and technical assistance contract, the contractor will work with counties to develop evaluation reports on several evidence-based childrens mental health services. The contractor will provide counties with technical assistance during the evaluation, and afterwards as part of quality improvement. During contract development, DHCS will continue to review deliverables related to county data reporting and performance outcomes evaluation with the Accountability Commission.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS continues to support and further the activities of the Accountability Commission's Evaluation Master Plan through ongoing data reporting system improvements, county assistance in data reporting, and data monitoring and reporting compliance efforts.

DHCS and the Accountability Commission continue to collaborate on various data workgroup efforts to help in creating uniform, standard data reporting and outcome evaluations.

DHCS continues to work with counties on their submission of mental health data. As part of the MHSA training and technical assistance contract, DHCS will work with the contractor and the Accountability Commission to develop training for counties on data reporting and performance outcomes evaluation in support of local and statewide performance outcome objectives and standards.

DHCS will meet with the Accountability Commission beginning in November 2013 to plan for the county guidance and training needed to effectively evaluate and report on the performance of MHSA funded programs.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: Health Care Services, Department of

To ensure that Health Care Services and other state entities can evaluate MHSA programs and assist the Accountability Commission in its efforts, Health Care Services should collect complete and relevant MHSA data from the counties.

Annual Follow-Up Agency Response From October 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve CSI and DCR data quality. Counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement, MHDA, has produced quarterly reports which contain the client counts, service counts, and percent error of data submissions to support monitoring and assistance for counties in improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created EPLD templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county data quality data. For the remainder of FY 2019-20, MHDA will train DHCS on use of the EPLD template, so county assistance can fully transition to DHCS staff. MHDA will continue to work closely with DHCS on CSI data monitoring and communicating with counties as needed. In FY 2020-21, MHDA and DHCS will continue to collaborate to transition CSI/DCR county technical assistance and training functions to DHCS staff. CSI corrective action plans will be created for counties and MHDA will conduct CSI data submission webinars for counties and provide technical assistance to vendors regarding business rules. For DCR, MHDA will provide additional training materials and quarterly trainings to counties in addition to data quality reports.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve the quality of the CSI and DCR data. Specifically, counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement, Mental Health Data Alliance (MHDA), has produced quarterly reports which contain the client counts, service counts, plus percent error of data submissions to further enable continued monitoring and assistance for counties and their progress towards improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created Enhanced Provide-Level Data (EPLD) templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county progress and compliance with submitting quality data. For the remainder of FY 2019-20, MHDA will provide training to DHCS regarding the use of the EPLD template so that assistance to counties can fully transition to DHCS staff. MHDA will also continue to work closely with DHCS in monitoring the improvement of CSI data and communicating with counties as needed.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Health Care Services is taking steps to collect complete and relevant MHSA data from the counties including Plans of Correction to address and correct data reporting errors. Health care Services is also working with a vendor to fully assume the vendor's role in monitoring county data.

Annual Follow-Up Agency Response From November 2018

DHCS entered into a vendor contract to improve data quality and completeness in the CSI and the MHSA DCR. The vendor created reports of findings (e.g., data entry errors, anomalies, missing data) and delineated the need for improved infrastructure and maintenance of data quality. The vendor has also provided customized technical

assistance (e.g., weekly webinars, monthly site visits) for counties and EHR vendors to improve data quality. DHCS is already seeing improvements in data quality. For years, client counts in CSI had been declining. Since the vendor began providing reports and technical assistance to the counties, the client counts have increased statewide by 7.6 percent for FY 2016-17 data. Since data quality is a continuous process, DHCS has extended the vendor's contract to FY 2018-19, and is negotiating a contract for FY 2019-20. Ultimately, DHCS plans to transition vendor-provided technical assistance and system training to DHCS staff within the contract timeframe for continued county support. This recommendation will be fully implemented in June 2020.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2017

DHCS contracted with a vendor to perform data clean up on the data included in the DHCS Client Services Information System and MHSA Data Collection and Reporting System. The purpose of the project is to provide DHCS, the MHSOAC, and the counties with accurate and complete data that may be used to track performance outcomes for Community Services and Supports, Full Service Partnership clients. This project is scheduled for completion over the next 12-18 months. With DHCS maintaining data quality thereafter. Counties have already received their comprehensive data error reports, and technical assistance calls to support the interpretation and use of these reports are currently underway. The data clean up project is to be completed by Fall 2018. Upon completion, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

DHCS is utilizing a contractor to help improve data reporting timeliness and data quality in the Client and Service Information system and the Data Collection and Reporting system.

Work on this project is will begin in FY 2016-17.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

All of the data that DHCS collects is relevant data. DHCS is working to improve the timeliness of the data that is reported and the completeness of the data that is reported. DHCS has implemented processes that have improved the timeliness of data reporting in both the Client and Services Information (CSI) data system and the Data Collection and Reporting (DCR) data system. Counties are now reporting data timely.

- Client and Services Information (CSI) Data: According to the DHCS September 2015 county submission status report, 50 counties are current in submitting. County submissions average 86% current with data submission.

- Data Collection Reporting (DCR): According to the DHCS September 2015 county submission status report, 57 counties are current in submitting DCR full service partnership data. County submissions average 97%.

DHCS is starting to focus on improving the completeness of the data submitted. Once DHCS is completed the work necessary to improve the completeness of data, this recommendation will be fully implemented.

DHCS is currently working on relevancy in its data modernization project. Once the data modernization project is done, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

DHCS continues to maintain monthly contact with county staff for obtaining status updates and providing technical assistance.

Client and Services Information (CSI):

According to the DHCS June 2014 county submission status report, 46 counties are current in submitting CSI data. The number of counties submitting late went from 9 counties in January 2014 to 12 counties in June 2014. Late submissions went from an average of 7 months delayed in January 2014, to an average of 3 months delayed by June 2014.

Data Collection Reporting (DCR):

According to the DHCS June 2014 county submission status report, 56 counties are current in submitting DCR full service partnership data. The number of counties submitting late decreased from 8 counties in January 2014 to 3 counties in June 2014. The number of counties that never submitted data decreased from 5 counties in January 2014 to 2 counties in June 2014. Late submissions went from 3 counties being an average of 8 months late in January 2014 to 1 county being 21 months late in June 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

For all counties late in reporting data, DHCS maintains monthly contact with county staff to obtain an update on their reporting status and timeline for becoming current with data reporting, and to offer any technical assistance. Submitted reports are attached to this document.

Client and Services Information (CSI): According to the DHCS January 2014 county submission status report, 49 counties are current in submitting CSI data. The number of counties submitting late decreased from 14 counties in September 2013 to 9 counties in January 2014. Late submissions went from an average of 10 months delayed in September 2013 to an average of 7 months delayed by January 2014.

Data Collection Reporting (DCR): According to the DHCS January 2014 county submission status report, 51 counties are current in submitting DCR full service partnership data. The number of counties submitting late decreased from 9 counties in September 2013 to 8 counties in January 2014. Late submissions went from an average of 15 months late in September 2013 to an average to 8 months late by January 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS continues to review county data submission for completeness and accuracy. According to the DHCS September 2013, county submission status reports, 44 counties are current in submitting Client and Services Information (CSI) data and 14 are not current in reporting. Regarding the Data Collection and Reporting (DCR) full service partnership data, 49 counties are current in submitting DCR data and 9 are not current.

To assist all counties in becoming fully compliant with their required reporting, DHCS established and operates a helpdesk for county data reporting questions; provides technical assistance to county and county vendor information technology staff; posts monthly county submission status reports on the DHCS website; and coordinates with CMHDA and the Accountability Commission for any needed system improvements and updates to data reporting requirements. In September 2013, DHCS began contacting the County Mental Health Director in each county without current data to request the county's reporting status and timeline for becoming current with data reporting.

DHCS will begin an in-depth assessment of CSI and DCR data completeness and relevancy for outcomes reporting purposes in the winter of 2013.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #7 To: Health Care Services, Department of

To ensure that Health Care Services and other state entities can evaluate MHSA programs and assist the Accountability Commission in its efforts, Health Care Services should resolve all known technical issues with the partnership and client services systems and provide adequate and expert resources to manage the systems going forward.

Annual Follow-Up Agency Response From November 2018

Department of Health Care Services migrated the Data Collection Reporting (DCR) System into the Behavioral Health Information Services System in October 2017. Migration for the Client Services Information System and DCR is fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From September 2016

DHCS migrated the Client and Service Information system into the new Behavioral Health Information Services system (BHIS) in April 2016. At this time, DHCS is not aware of any technical issues. DHCS is also planning to migrate the Data Collection and Reporting system into the BHIS and expects to complete the migration during 2017.

This recommendation will be fully implemented when the DCR migration is completed

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

DHCS has resolved the technical problems counties are experiencing in uploading data as evidenced by the improvements in timely reporting. DHCS is also updating the CSI system that should improve data submission. The upgrades to the CSI system should be completed by the end of 2015. Finally, DHCS is continuing to work on the data modernization project, but full implementation of this recommendation does not depend upon fully implementing the data modernization project.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

As of March 3, 2014, all of the eight Data Collection and Reporting (DCR) modifications were completed in accordance with the contract timeframe. In addition, DHCS is in the process of creating a plan to update the technology platform of the Client and Services Information (CSI) system in an effort to improve data integrity.

DHCS Information Technology staff continues to work closely with county staff to resolve technical issues in the CSI and DCR systems. Additionally, bi-monthly meetings are held with the Mental Health Services Oversight and Accountability Commission to plan for data strengthening efforts for CSI and DCR data and to coordinate data system improvements.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

As of January 2014, six of the eight Data Collection and Reporting modifications have been completed. This project is expected to be completed by March 2014, in accordance with the contract timeframe.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS Information Technology staff continue to work directly with county staff and their system vendors to resolve county data submission issues.

Through collaboration with the Accountability Commission, DHCS has received additional resources to assist with implementing system updates to the Data Collection and Reporting full service partnership data system. System updates necessary to address priority system improvements began in July 2013 and are to be completed by March 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #8 To: Health Care Services, Department of

Health Care Services should, as soon as is feasible, revise or create a reasonable and justifiable allocation methodology to ensure that counties are appropriately funded based on their identified needs for mental health services. Health Care Services should ensure that it reviews the methodology regularly and updates it as necessary so that the factors and their weighting are appropriate.

Annual Follow-Up Agency Response From October 2015

DHCS has revised the allocation methodology. The uninsured population is no longer included in the estimate of need because the Affordable Care Act is expected to insure most Californians. The data in the remaining factors have been updated to calendar year 2015. DHCS continues to assess the methodology annually.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

DHCS met with the County Behavioral Health Director's Association, the Mental Health Services Oversight and Accountability Commission, the California Mental Health Planning Council, and Mental Health America in California in April 2014 to discuss the current allocation methodology. DHCS decided to maintain the current MHSA county allocation ratios for the 2014-2015 fiscal year to allow more time for review and discussion; and address the allocation methodology in greater detail for the 2015-16 fiscal year distributions.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

Discussing the allocation methodology with stakeholders is important; however, because DHCS has neither revised nor justified the current methodology, this recommendation is not fully implemented.

6-Month Agency Response

DHCS has reviewed the current allocation methodology and supporting data sources for the distribution of MHS funds. During spring 2014, DHCS will engage various stakeholders to gather input on the allocation distribution methodology. When finalized, DHCS will issue an Information Notice containing the MHSA allocation methodology for FY 2014-15 by June 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS is reviewing the current allocation methodology for the distribution of MHS funds and updating the data sources to determine what impact the updated data sources will have on the current distribution percentages. DHCS is also reviewing the allocation methodology used for the 2011 Realignment Behavioral Health Services Growth Special Account funds. By mid-October 2013, DHCS will obtain stakeholder input on the allocation methodology for the 2011 Realignment growth funds. The information gathered from this effort will be leveraged, where applicable, as DHCS works in consultation with the Accountability Commission and CMHDA in reviewing the current MHSA allocation methodology and determining any updated factors or weighting based on needs for mental health services. DHCS will also seek stakeholder input on the allocation factors. Finally, DHCS will issue an Information Notice containing the MHSA allocation methodology for FY 2014-15 by June 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #9 To: Mental Health Services Oversight and Accountability Commission

To ensure that counties have needed guidance to implement and evaluate MHSA programs, the Accountability Commission should issue regulations, as appropriate, for Prevention and Innovation programs.

Annual Follow-Up Agency Response From January 2016

The Office of Administrative Law approved the Innovation regulations on July 16, 2015 and the Prevention and Early Intervention regulations on October 7, 2015. The Innovation regulations became effective on October 1, 2015 and the Prevention and Early Intervention regulations became effective on October 6, 2015.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From September 2015

On June 5, 2015, the Mental Health Services Oversight and Accountably Commission submitted the final rulemaking for both the Innovation and Prevention and Early Intervention (PEI) regulations to the Office of Administrative Law. On July 16, 2015 the Office of Administrative Law approved the Innovation regulations which will become effective on October 1, 2015. On July 17, 2015 the Office of Administrative Law disapproved the PEI regulations due to a minor inconsistency in one of the definitions. The Commission made the necessary changes to the PEI regulation section to address the Office of Administrative Law's decision and on August 31, 2015, resubmitted the revised PEI rulemaking file to the Office of Administrative Law. The Office of Administrative Law has until October 13, 2015 to render its decision.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2014

The regulatory process for Prevention and Innovation programs has begun but is not yet complete. The Mental Health Services Oversight and Accountably Commission's notice of proposed regulations for Prevention and Early Intervention (PEI) programs was published in the California Regulatory Notice Register No. 23-Z on June 6, 2014. The 45-day public comment period on this proposed rulemaking closed on July 24, 2014 and the Commission held a public hearing on July 24, 2014 to receive public comments. The Commission voted at its August, September, and October 2014 meetings to revise the language of the proposed PEI regulations in response to comments received during the 45-day public comment period. The revisions required 15-day public comment periods. The Commission, at its December 18, 2014, meeting is scheduled to respond to the comments from these 15-day comment periods. The Commission anticipates submitting the final rulemaking file to the Office of Administrative Law in February or March 2015.

The notice of proposed regulations for Innovation programs was published in the California Regulatory Notice Register No. 28-Z on July 11, 2014. The 45-day public comment period on this proposed rulemaking closed on August 28, 2014 and the Commission held a public hearing on August 28, 2014 to receive public comments. The Commission voted at its October 23, 2014 meeting to revise the language of the proposed Innovation regulations in response to comments received during the 45-day public comment period. The revisions required a 15-day public comment period. The Commission at its December 18, 2014, meeting is scheduled to respond to the comments from the 15-day comment period. The Commission anticipates submitting the final rulemaking file to the Office of Administrative Law in February 2015.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The regulatory process for Prevention and Innovation programs is almost fully implemented. The Mental Health Services Oversight and Accountably Commission's proposed regulations for Prevention and Early Intervention (PEI) programs were published in the California Regulatory Notice Register No. 23-Z on June 6, 2014. The 45-day public comment period on this proposed rulemaking closed on July 24, 2014 and the Commission held a public hearing on July 24, 2014 to receive public comments. The Commission is scheduled to vote at its August 28, 2014 meeting on whether to revise the language of the proposed PEI regulations in response to comments received during the public comment period. Additional public comment period will be required if revisions are made to the proposed PEI regulations.

The proposed regulations for Innovation programs were published in the California Regulatory Notice Register No. 28-Z on July 11, 2014. The 45-day public comment period on this proposed rulemaking will close on August 28, 2014 and the Commission will hold a public hearing on August 28, 2014 to receive public comments. The Commission is scheduled to vote at its September 25, 2014 meeting on whether to revise the language of the proposed Innovation regulations in response to comments received during the public comment period. Additional public comment period will be required if revisions are made to the proposed Innovation regulations.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The regulatory process for Prevention and Innovation programs has begun but is not yet complete. For details see response to Recommendation #10.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The regulatory process for Prevention and Innovation programs has begun but is not yet complete.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #10 To: Mental Health Services Oversight and Accountability Commission

To ensure that counties have needed guidance to implement and evaluate MHSA programs, the Accountability Commission should commence the regulatory process no later than January 2014.

1-Year Agency Response

The Commission commenced the pre-regulatory stakeholder input process for both Prevention and Early Intervention and Innovation programs in August of 2013. The official regulatory process for the Prevention and Early Intervention programs was commenced on June 6, 2014. (See California Regulatory Notice Register No. 23-Z published on June 6, 2014) The official regulatory process for the Innovation programs was commenced on July 11, 2014. (See California Regulatory Notice Register No. 28-Z published on July 11, 2014)

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Commission commenced the process but has not yet completed it. On August 8, 2013, the Commission convened a Prevention and Early Intervention (PEI) and Innovation (INN) Regulation Workgroup. Input from the Workgroup was used to devleop a first draft of the proposed regulations, which was reviewed by the Workgroup on 9/20/13. The Commission met on 10/24/13 to discuss the first draft of the proposed regulations and to recieve public comment. The public comment was incorporated into the next draft and on 11/21/13 the Commission met and adopted the draft regulations and authorized the Executive Director to submit them to the Office of Adminstrative Law. The Commission is in the process of drafting the Initial Statement of Reasons.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

On August 8, 2013, the Commission commenced the regulatory process with a Prevention and Early Intervention (PEI) and Innovation (INN) Regulation Work Group meeting, which reviewed principles for regulations and made suggestions on which sections of PEI and INN statute needed to be clarified. This feedback was incorporated into a first draft of regulations, which was reviewed by the Work Group on 9/20. Also, the Commission reviewed the principles on 9/26 and will meet on 10/24 to discuss the first draft.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

This recommendation is assessed as partially implemented because the Accountability Commission is just drafting regulations. Until the Accountability Commission has completed the regulatory process the recommendation cannot be assessed as fully implemented.

Recommendation #11 To: Mental Health Services Oversight and Accountability Commission

To fulfill its charge to evaluate MHSA programs, the Accountability Commission should undertake the evaluations specified in its implementation plan.

Annual Follow-Up Agency Response From November 2017

The five-year Evaluation Master Plan adopted by the Commission in 2013 served well as an initial strategic guide for research and evaluation. The Implementation Plan that accompanied the Evaluation Master Plan was designed to be a living document to allow for adding and revising the list of evaluations over the timeline of the plan in response to environmental events. The Commission in 2015 contracted with University of California, Davis and University of California, Los Angeles to conduct the last two studies listed on the Implementation Plan for FY 2017/18, the fifth year of the five-year plan. The UC Davis contract to look at early psychosis was completed in June of 2017. The last deliverable (fact sheets) on the UC Los Angeles contract looking at the effectiveness of programs for older adults is due December 30, 2017.

The initial phase is now completed. The Commission is working to develop metrics for monitoring outcomes, tracking reductions in disparities and developing tools to support broader public accountability, each of which are referenced in the MHSA. The MHSA outlines specific goals including reductions in unemployment, homelessness, school failure, out of home placements, suicide, criminal justice involvement, and prolonged suffering. The Commission is working on developing strategies to measure progress against those goals. Such work includes conducting policy research projects similar to the recently completed project on the intersection of the criminal justice and mental health systems that evaluated the challenges and opportunities to reducing the number of adults with mental health needs in the criminal justice system and improving outcomes for those who are in custody. These projects that integrate evaluation and data analyses will be identified and prioritized by the Commission through an open and deliberative process.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From September 2015

The Evaluation Master Plan, adopted by the Commission on March 2013, is a five year plan and will not be fully implemented until fiscal year 2017-2018. Since August 2013 when the BSA issued this recommendation, the Commission has undertaken the evaluations specified in the Commission's implementation plan for this period of time (i.e. fiscal years 2013-2014 and 2014-2015). At this point in time, two months into the 2015-16 fiscal year, the evaluations for the current fiscal year have not yet begun. However, the Commission followed its normal process and at the October 23, 2014 meeting voted on specific evaluation activities to take place in fiscal year 2015-2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2014

The Evaluation Master Plan, adopted by the Commission on March 2013, is a five year plan and will not be fully implemented until fiscal year 2017-2018. Since August 2013 when the BSA issued this recommendation, the Commission has undertaken all of the evaluations specified in the Commission's implementation plan for this period of time (i.e. fiscal years 2013-2014 and 2014-2015). At its October 23, 2014 meeting the Commission voted on the specific evaluation activities to take place in fiscal year 2015-2016.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The Evaluation Master Plan, adopted by the Commission on March 2013, is a five year plan and will not be fully implemented until fiscal year 2017-2018. Since August 2013 when the BSA issued this recommendation, the Commission has undertaken all of the evaluations specified in the Commission's implementation plan for this period of time (i.e. fiscal years 2013-2014 and 2014-2015) and has begun to plan for evaluation activities to take place in fiscal year 2015-2016.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The Evaluation Master Plan, adopted by the Commission on March 2013, is a five year plan and will not be fully implemented until fiscal year 2017-2018. Since August 2013 when the BSA issued this recommendation, the Commission has undertaken all of the evaluations specified in the Commission's implementation paln for this period of time (i.e., fiscal year 2013-2014) and has begun to plan for evaluation activities to take place in fiscal year 2014-2015.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The Evaluation Master Plan, adopted by the Commission in March 2013, is a five year plan and will not be fully implemented until fiscal year 2017-2018. Since the BSA issued this recommendation, the Commission has undertaken the evaluations specified in the implementation plan for this period of time. The Commission has:

a. Continued progress on the UCLA contract for priority indicators

b. Continued progress on the UCD contract for reducing disparities in access to care

c. Continued progress on the UCLA contract for PEI

d. Continued progress on the RDA contract for the community program planning

e. Entered into a contract with CiMH for INN evaluations

f. Entered into a contract with MHDA for CSI data collection and reporting strengthening and support

g. Issued a contract with MHDA and DHCS for IT strengthening of the DCR, which is owned by DHCS

h. Began refining the measurements of existing indicators

i. Began negotiating a contract to determine effectiveness of methods for engaging and serving TAY clients

j. Issued an RFP to develop a system to track outcomes for persons in less intensive services than FSP

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #12 To: Mental Health Services Oversight and Accountability Commission

To ensure that it can fulfill its evaluation responsibilities, the Accountability Commission should examine its prioritization of resources as it pertains to performing all necessary evaluations.

6-Month Agency Response

The Commission has examined its prioritization of resources and concludes that it will be able to perform all necessary evaluations stated in the Evaluation Master Plan and Implementation Plan since funding for the Evaluation Master Plan was included in the Governor's Fiscal Year 2013-14 Budget.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The Commission has examined its prioritization of resources and concludes that it will be able to perform all necessary evaluations stated in the Evaluation Master Plan and Implementation Plan since funding the Evaluation Master Plan was included in the Governor's FY 2013-14 May Revision.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Accountability Commission cites that funds for its evaluation efforts are included in the Governor's FY 2013-14 May Revision. Because the May Revision is a proposal rather than the approved budget, the Accountability Commission needs to examine its prioritization of resources in light of the approved budget.

Recommendation #13 To: Mental Health Services Oversight and Accountability Commission

To report on the progress of MHSA programs and support continuous improvement, the Accountability Commission should fully use the results of its evaluations to demonstrate to taxpayers and counties the successes and challenges of these programs.

6-Month Agency Response

Following the BSA recommendation, the Commission utilzied its Evaluation Committee to review the results of all the final evaluation reports:UCLA Phase III Final Report on September 4, 2013 and the UCLA PEI Deliverable 1 Report on February 4, 2014. In addition, the Evaluation Committee discussed how best to achieve this objective at its October 24, 2013 meeting. The Commission will continue to use this process with the future evaluation reports generated pursuant to the evaluation implementation plan for FY 2013/14 through 2017/18.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

Following the BSA recommendation, the Commission utilized its Evaluation Committee to review the results of evaluations, including the UCLA Phase III Final Report on September 4, 2013. The Commission will continue to use this process with future evaluation deliverables.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Although the Accountability Commission substantiated its reivew of the UCLA Phase III Final Report, this is a single event. The recommendation addresses our concern that the Accountability Commission ensure that it fully uses the results of all of its evaluations,

Recommendation #14 To: Mental Health Planning Council, California

The Planning Council should take steps to ensure that it annually reviews the overall effectiveness of MHSA programs in accordance with state law.

1-Year Agency Response

The Planning Council has released the Data Notebook to all the county mental health boards for their use in reporting their review of program outcomes for their county to the Council. The Notebooks are due July 31, 2014 although some counties have asked for an extension. Additionally, the Council has begun holding public forums around the state to obtain input from stakeholders at the local level regarding the various changes in the mental health delivery system over the past several years. The first forum was held in May in Merced, in San Bernardino in July and a 3rd is scheduled for early September in Yolo County. The Council plans to hold mono-lingual forums with interpreters, also to reach out to native communities and the LGBTQ communities for designated forums to receive their input. Lastly, the Council is working with the CA Behavioral Health Directors Association to finalize the protocol and format for program reviews of children's programs in small counties. It is anticipated that the reviews will begin in late 2014/early 2015. Aggregated reports from the Data Notebook and Public Forum projects are expected to be released late 2014/early 2015.

Until county-reported data becomes available for the Council to use in fulfilling its responsibilities to review and report on program performance, we will continue to use these optional methods of gathering program performance data at the local level. These projects have been launched so this Audit Review item is fully implemented.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

Over the past months, the Council continued to meet regularly with both the DHCS and the MHSOAC and the group has identified some possibilities to address the need for an effective data collection and outcomes reporting system. Necessary research, testing and documentation are underway as initial steps to future implementation. Meanwhile, the Council is in the process of implementing several activities over this year to obtain input at the local level regarding the effectiveness of mental health services, including MHSA-funded services, through public forums, a data notebook project with the local mental health boards and also on-site county program reviews as alternative ways to fulfill the above mandate until there are sufficient sources of outcomes and data for the Council to review.

We are currently developing the data notebook and training curriculum, identifying the format and topic areas for the public forums, and designing the review protocol for the on-site program reviews. These will be pilot tested in the Spring and early Summer 2014. Final reports are expected by the end of the year.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

In its response to the Auditor's findings, the Council reached out to DHCS and the MHSOAC in August, requesting a meeting to explore avenues of collaboration on the mutual mandates. The three entities met in mid-October to discuss their strategies to meet these recommendations, and will continue to meet on a monthly basis to identify and define individual roles and tasks as outlined in WIC 5610 sec 120 (a) and (b).

In addition to the external workgroup with DHCS and MHSOAC, the Council has convened an internal workgroup to better understand the process of evaluation- the available data sources, the criteria for good reporting (i.e. pertinent, accessible, and informative performance indicators, which the Council is mandated to approve) and the critical review of the reports that are generated. When completed, this information will be shared with the entire Council in order to clarify the expectations of the mandate and comply with the Audit's recommendation. (This workgroup is on-going, with no established end-date at this time).

Continuing the development of the new Data Notebook for use by county Mental Health Boards in reporting on mental health access and service provision by age, gender, and ethnicity relative to the county population using quality of life indicators such as incarceration and hospitalization rates, high school graduation, educational persistence, and employment. We have a target date of March 1, 2014 for this.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #15 To: Mental Health Planning Council, California

The Planning Council should document and make public the reviews that it performs of MHSA programs to demonstrate that it is performing all required reviews.

Annual Follow-Up Agency Response From September 2015

The California Mental Health Planning Council released 4 reports in early 2015 which document the Council's reviews of the public mental health system including MHSA programs. The Council has proactively sought data and information from a variety of sources and endeavored to solicit input and viewpoints from specific populations. These reports fulfill the requirement that the Council make public the reviews it performs.

Additional reports will be released each January of reviews and work done in the prior year. The reports are posted on the Council's website, distributed electronically and shared with key policy makers.

Copies of these reports will be sent to the State Auditor via email.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

As mentioned in the previous response, the Council has launched several projects to collect data and input on program performance from the local level. Reports on the information collected are expected to be released in late 2014/early 2015. Additionally, as the projects continue, additional reports will be released throughout the year. All reports will be posted on the Council's website.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

As mentioned in the previous recommendation, several review activities are in process and final reports are expected at the end of the year. In addition to those reports, the Council anticipates releasing smaller reviews, recommendations and white papers throughout the year.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Council instituted and awarded its inaugural Joe Mortz Memorial "Spotlight on Excellence" in Community mental health commendation. CMHPC solicited nominations statewide for community mental health programs that reflected the values and principles of MHSA in its operations, outreach, and services, although it was not required that they be funded through MHSA. Scores were based on understanding and adherence to the principles, sustainability, cost-per-consumer/partner, and benefit to community. The "Peers Empowering Peers" project of Napa County was identified as the most closely aligned with MHSA principles and practice. The congratulation letter is posted on the CMHPC website, and letters describing the program were sent to local, state, and national legislators in order to highlight the success within their districts. The same letters were distributed electronically to advocates and supporters that included Mental Health Boards and Commissions, MHSA coordinators, and the CMHDA.

[WIC 5772- (a), (b), (c)(4), (e).]

California State Auditor's Assessment of 60-Day Status: Partially Implemented

The Planning Council's response provides a single instance for which it publicized its activities. The recommendation envisions that the Planning Council have a process to ensure that it documents and makes public all reviews it performs.

Recommendation #16 To: Health Care Services, Department of

To improve the quality of county processes related to measuring program performance, Health Care Services should use its performance contracts with counties to ensure that they specify MHSA program goals in their plans and annual updates and include those same goals in their contracts with program providers.

1-Year Agency Response

The FY 2014-15 Performance Contract provides that county contracts with providers shall include the performance goals from the County's three-year program and expenditure plan and annual updates.

DHCS will audit the counties for compliance as part of program reviews. DHCS continues to draft regulations to ensure MHSA-funded contract providers have contractual goals that are consistent with the approved three-year program and expenditure plans and annual updates.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Performance Contract provides that county contracts with providers shall include the performance goals from the Countys three-year program and expenditure plan and annual updates that apply to each providers programs and services.

DHCS continues to draft regulations to ensure MHSA-funded contract providers have contractual goals that are consistent with the approved three-year program and expenditure plans and annual updates. After regulations are promulgated, DHCS will audit the counties for compliance.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The Performance Contract provides that county contracts with providers shall include the performance goals from the County's three-year program and expenditure plan and annual updates that apply to each provider's programs and services. The Performance Contract will be sent to the counties for signature approval in September 2013. The time required to fully execute this contract will depend on how quickly each county is able to gain Board of Supervisor approval. In the next report, this recommendation's implementation status will be updated with the number of counties that have fully executed their Performance Contract.

DHCS is also drafting regulations to ensure MHSA-funded contract providers have contractual goals that are consistent with the approved three-year program and expenditure plans and annual updates. Draft regulations will be made available for public comment in the spring of 2014. We anticipate that final regulations will be adopted during the fall of 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #17 To: Health Care Services, Department of

To improve the quality of county processes related to measuring program performance, Health Care Services should use its performance contracts with counties to ensure that they identify meaningful data to measure the achievement of all their goals, set specific objectives, and require their program providers to capture those data so they can use the data to verify and report the effectiveness of their MHSA programs.

1-Year Agency Response

The FY 2014-15 Performance Contract contains language specifying that counties must: report required data for the purpose of evaluating mental health outcomes; include reports on the achievement of performance outcomes in their three-year program and expenditure plan and annual updates; and, must include the performance goals from the three-year program and expenditure plan and annual updates in contracts with providers.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Performance Contract contains language specifying that counties must: report required data for the purpose of evaluating mental health outcomes; include reports on the achievement of performance outcomes in their three-year program and expenditure plan and annual updates; and, must include the performance goals from the three-year program and expenditure plan and annual updates in contracts with providers.

DHCS continues its ongoing effort to identify outcomes/measures for children and youth as part of the Performance Outcomes System Plan for Medi-Cal Specialty Mental Health Services for Children and Youth (POS). Since some of the evaluation efforts in the POS and the Evaluation Master Plan are similar, DHCS and the Accountability Commission will continue to collaborate to develop program performance measurements.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The Performance Contract contains language specifying that counties must: report required data for the purpose of evaluating mental health outcomes; include reports on the achievement of performance outcomes in their three-year program and expenditure plan and annual updates; and, must include the performance goals from the three-year program and expenditure plan and annual updates in contracts with providers.

Specific statewide outcomes are established jointly by DHCS and the Accountability Commission, in collaboration with CMHDA and in consultation with the California Mental Health Planning Council (CMHPC). As additional outcomes are needed, DHCS will continue to work with the Accountability Commission and the CMHPC to develop consistent outcome definitions and uniform, statewide data reporting requirements by leading and/or consulting with various performance outcomes committees and workgroups in support of the Evaluation Master Plan.

The Performance Contract will be sent to the counties for signature approval in September 2013. The time required to fully execute this contract will depend on how quickly each county is able to gain Board of Supervisor approval. In the next report, this recommendation's implementation status will be updated with the number of counties that have fully executed their Performance Contract.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #18 To: Health Care Services, Department of

Health Care Services should develop standardized data collection guidelines or regulations, as appropriate, that will address inconsistencies in the data that counties report to the State. In developing these guidelines or regulations, Health Care Services should consult with the Accountability Commission to ensure that data collected reasonably fulfill statewide evaluation purposes.

Annual Follow-Up Agency Response From October 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve CSI and DCR data quality. Counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement, MHDA, has produced quarterly reports which contain the client counts, service counts, and percent error of data submissions to support monitoring and assistance for counties in improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created EPLD templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county data quality data. For the remainder of FY 2019-20, MHDA will train DHCS on use of the EPLD template, so county assistance can fully transition to DHCS staff. MHDA will continue to work closely with DHCS on CSI data monitoring and communicating with counties as needed. In FY 2020-21, MHDA and DHCS will continue to collaborate to transition CSI/DCR county technical assistance and training functions to DHCS staff. CSI corrective action plans will be created for counties and MHDA will conduct CSI data submission webinars for counties and provide technical assistance to vendors regarding business rules. For DCR, MHDA will provide additional training materials and quarterly trainings to counties in addition to data quality reports.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve the quality of the CSI and DCR data. Specifically, counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement and MHDA has produced quarterly reports which contain the client counts, service counts, plus percent error of data submissions to further enable continued monitoring and assistance for counties and their progress towards improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created EPLD templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county progress and compliance with submitting quality data. For the remainder of FY 2019-20, MHDA will provide training to DHCS regarding the use of the EPLD template so that assistance to counties can fully transition to DHCS staff. MHDA will also continue to work closely with DHCS in monitoring the improvement of CSI data and communicating with counties as needed.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Health Care Services is taking steps to improve and standardize the data it collects from counties.

Annual Follow-Up Agency Response From November 2018

DHCS contracted with a vendor to perform data clean up on the data included in the DHCS CSI and MHSA DCR. The purpose of this project is to provide DHCS, the MHSOAC, and the counties with accurate and complete data that may be used to track performance outcomes for Community Services and Supports, Full Service Partnership clients. In addition, the contract will develop templates and provide technical assistance to counties regarding data collection to ensure that data is being submitted consistently. Counties have already received comprehensive data error reports, and technical assistance calls to support the interpretation and use of these reports are currently underway. The project will be completed by December 2020. DHCS sent corrective action plan letters to counties in October 2018 identifying the errors for FY 2015-16 and 2016-17 that need to be corrected and to ask the counties to submit a corrective action plan to correct those errors and to explain processes they will implement to ensure quality data going forward.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From October 2015

DHCS is planning to meet with the Mental Health Services Oversight and Accountability Commission (MHSOAC) to review the current guidance provided to counties regarding data collection and reporting to the DCR to determine the need to develop more standardized guidance. DHCS will develop more standardized guidance if it determines in consultation with MHSOAC that such guidance is needed.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

As part of the review and development of MHSA regulations, DHCS continues to consult with the Accountability Commission to determine any regulatory changes needed to ensure data are collected consistently and in support of the Evaluation Master Plan.

If needed, draft regulations will be developed in consultation with the Accountability Commission and made available for public comment in the spring of 2015, with final regulations adopted during the fall of 2016.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

DHCS continues to work with the Accountability Commission on the development of MHSA regulations. As part of the review and development of MHSA regulations, DHCS continues to consult with the Accountability Commission to determine any regulatory changes needed to ensure data are collected consistently and in support of the Evaluation Master Plan. Given the significant time required to develop the audit regulations, the development of the data collection regulations have been delayed. DHCS expects to begin developing regulations during the summer of 2014 with final regulations to be adopted beginning in the winter of 2014.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

DHCS will consult with the Accountability Commission to determine if regulation changes are necessary to ensure data are collected consistently and in support of the Evaluation Master Plan. If needed, draft regulations will be made available for public comment in the spring of 2014. We anticipate that final regulations will be adopted during the fall of 2014.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #19 To: Health Care Services, Department of

To help ensure county compliance with stakeholder regulations, Health Care Services should provide technical assistance to counties on the MHSA local planning review process and ensure that its guidance to counties is clear and consistent with state regulations.

Annual Follow-Up Agency Response From August 2019

Implementation of the MHSA program reviews commenced January 2019. The review protocol includes a review of the county's community program planning process along with technical assistance provided by DHCS while at the site. A Performance Contract Review report details any findings, recommendations, and suggested improvements found during the review. The county has 60 days after receiving the Performance Contract Review report to submit a corrective action plan.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Through its periodic program reviews, Health Care Services provides counties with technical assistance on the MHSA local planning review process.

Annual Follow-Up Agency Response From November 2018

DHCS developed a protocol (evaluation tool) and review program and performed pilot reviews beginning in February 2018. The review protocol includes a review of the county's community program planning process. After each county review, DHCS will prepare a report of findings. Any county with findings will need to develop a corrective action plan. DHCS will approve and monitor the corrective action plan. DHCS piloted the review protocol in four counties and will fully implement the review protocol in January 2019.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2015

DHCS is planning to include training and technical assistance on the local planning in the California Institute for Behavioral Health Solutions (CIBHS) contract in FY 2015-16. Once that is complete, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The contract containing funding for MHSA training and technical assistance was fully executed in June 2014. The contract continues to include training to Local Mental Health Boards on their role in reviewing MHSA expenditure plans and annual updates and implementing the MHSA. Additional work orders will be developed during FY 2014-15 for county training specific to the stakeholder process.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

Funding for training to counties on the MHSA local planning review process is included in the MHSA training and technical assistance contract. A detailed work order for this deliverable is in development. The contract will also provide training to Local Mental Health Boards on their role in implementing the MHSA. While DHCS expected to fully execute the contract in October, the contract was delayed due to necessary revisions to the Scope of Work. DHCS expects the contract to be fully executed during February 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS oversees the MHSA training and technical assistance contract that provides a variety of training options and technical assistance to county mental health plans and service providers. As part of this contract, DHCS will continue to work with the contractor and the Accountability Commission to develop training for counties on the MHSA local planning review process. This contract also provides training to Local Mental Health Boards on their role in implementing the MHSA. DHCS expects to finalize the contract terms in September 2013 and that the contract will be fully executed in October 2013.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #20 To: Santa Clara County Mental Health Department

Santa Clara County should review its existing MHSA contracts and by December 31, 2013, or as soon as is feasible, amend them as necessary to include plan goals.

Annual Follow-Up Agency Response From October 2014

Please refer to our one-year response.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

Per the County's February 2014 update response, the County has incorporated plan goals and performance measure requirements into FY15 MHSA contracts. The program description section of service contracts (Exhibit A) has been redesigned to clearly specify program goals, objectives, process measures, outcomes to be achieved and the methods/approaches that will be utilized to evaluate if outcomes were achieved. The goal(s) listed in an MHSA contract is/are linked to the MHSA component level goal(s) reflected in the County's MHSA plan. This process has been completed and contracts are in final stages of execution for FY15.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The County continues to collaborate with the three other counties audited, Los Angeles County, Sacramento County, and San Bernardino County on creating a framework/tool incorporating plan goals in MHSA funded contracts. We are all in the process of testing the tool and applying it to MHSA funded contracts. Attached is the current draft of the tool. The Counties plan to share the work we have done on addressing this audit recommendation and provide an overview of the conceptual model/tool with our sister counties. The process will also involve vetting the tool with the Countys MHSA contracted providers and local stakeholders. The County will be incorporating the plan goals and performance measure requirements into the Countys FY2014-15 MHSA contracts; and anticipates completing the work related to the audit recommendation by July 1, 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

The County is currently working on a creating a framework incorporating plan goals in MHSA funded contracts in collaboration with the three other counties audited, Los Angeles County, Sacramento County, and San Bernardino County. All are in agreement that a common approach to the audit recommendation would align with the broader concept regarding consistent statewide approach to evaluation. The four counties participated in a teleconference on September 30, 2013 and agreed to develop a common approach to the determination of MHSA program goals and performance measures. A second teleconference took place on October 8, 2013 to outline the steps to MHSA program goal identification. The work is conceived to be conducted in three phases. The first phase involves the creation of a conceptual model to be developed in collaboration with the four audited counties; the second phase will involve the application of the model by each county to their MHSA contracts; and the third phase will involve each county executing contract amendments as needed to their MHSA contracts. Santa Clara County anticipates implementing the conceptual model into our FY2014-15 contract amendments. The process will include input on the proposed conceptual model from the County staff, MHSA stakeholders, and the Mental Health Board. To that end, the County MHD Director reported the proposed approach to the County's Performance Quality Improvement Committee (PQIC) on October 3, 2013. The PQIC, comprised of 40+ County and Contract Provider representatives, responded favorably to the proposed three phased approach to the audit recommendation.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #21 To: Santa Clara County Mental Health Department

Santa Clara County should ensure that all MHSA invoices are adequately supported with information that demonstrates that MHSA services were provided.

Annual Follow-Up Agency Response From September 2015

Santa Clara County has fully implemented the recommendation. Since our last reporting status update provided in October 2014 the County has created another set of invoicing guidelines and templates specifically for the current fiscal year (FY) 2016. The new FY16 invoicing templates includes invoicing guidelines, information about invoice types, and details about eligible services. Attached is supporting documentation that demonstrates the work the County has done around the implementation of this particular recommendation. Annually, the County will continue to review invoicing guidelines: update guidelines and templates as needed each fiscal year. In addition, the County will continue to provide one-on-one trainings to our contracted providers around the County's invoicing guidelines and use of templates.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2014

The County has drafted invoicing guidelines for different types of invoicing, e.g., direct service, outreach, flex/housing, cost-reimbursement, and dedicated beds. Initially, the County had planned on sharing the guidelines with the Behavioral Health Contractors' Association (BHCA) of Santa Clara County and all our contract providers in August 2014 but has been delayed to November 2014.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

The County has drafted invoicing guidelines for different types of invoicing, e.g., direct service, outreach, flex/housing, cost-reimbursement, and dedicated beds. The County will be sharing the guidelines with the Behavioral Health Contractors' Association (BHCA) of Santa Clara County and all our contract providers in August 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

The County is standardizing and documenting invoicing procedures for each program and type of invoice (e.g. direct services, indirect outreach services, flex funds, dedicated bed contracts, etc). The invoicing documentation will address invoice formats as well as supporting documentation requirements. The County anticipates completing the work by July 1, 2014 and the procedures will be circulated to the affected contract providers.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

As part of the MHSA process described for recommendation #1, the County will conduct a review of billing and invoicing procedures of each of the MHSA contracts. This process will include identifying documentation requirements for each category of invoices, i.e., direct services, flex funds, etc, and will establish invoicing requirements for each invoice category. Those requirements will be standardized and included in contract amendments slated for FY2014-15.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #22 To: Sacramento Department of Health and Human Services, County of

Sacramento County should review its existing MHSA contracts and by December 31, 2013, or as soon as is feasible, amend them as necessary to include plan goals.

1-Year Agency Response

Sacramento County has fully implemented the recommendation. As previously reported, the County applied the framework/tool developed collaboratively with the three other audited counties (Los Angeles, San Bernardino, and Santa Clara) to ensure that plan goals are incorporated into MHSA funded contracts. Any necessary contract revisions/amendments have been incorporated into the Fiscal Year 2014-15 contract scopes and those contracts are now in the final stages of execution with a contract term beginning July 1, 2014.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

Sacramento County has completed the internal review of goals as stated in the Mental Health Services Act (MHSA) plans as compared with the goals captured in contracts for MHSA-funded programs.

As previously reported, the County has worked collaboratively with the three other audited counties (Los Angeles, San Bernardino, and Santa Clara) to develop a framework/tool to ensure plan goals are incorporated into MHSA funded contracts. This tool is now complete and attached for your reference. The County is now applying this tool to the applicable MHSA funded contracts in order to execute necessary contract revisions/amendments for the upcoming contract year. Sacramento County anticipates completion of this entire process with updated Fiscal Year 2014-15 scopes by June 30, 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

Sacramento County is in the process of implementing the recommendation and as such, has begun the internal review of goals as stated in the Mental Health Services Act (MHSA) plans as compared with the goals captured in contracts for MHSA-funded programs. Simultaneously, the County is participating in creating a framework incorporating plan goals in MHSA funded contracts in collaboration with the three other audited counties (Los Angeles, San Bernardino, and Santa Clara). All are in agreement that a common approach to the audit recommendation would also align with the broader concept of a consistent statewide approach to evaluation. The four counties participated in a teleconference on September 30, 2013 and agreed to explore a common approach to the determination of MHSA program goals and performance measures. A second teleconference took place on October 8, 2013 to outline the next steps to MHSA program goal identification. The work is conceived to be conducted in three phases. The first phase involves the creation of a conceptual model to be developed in collaboration with the four audited counties; the second phase will involve the application of the model by each county to their MHSA contracts; and the third phase will involve each county executing contract revisions/amendments as needed to the MHSA contracts. Sacramento County anticipates completion of this entire process with updated Fiscal Year 2014-15 scopes capturing the plan goals prepared for inclusion in MHSA-funded contracts by June 30, 2014.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation #23 To: San Bernardino Department of Behavioral Health Administration, County of

San Bernardino County should review its existing MHSA contracts and by December 31, 2013, or as soon as is feasible, amend them as necessary to include plan goals.

1-Year Agency Response

The County of San Bernardino Department of Behavioral Health (DBH) has implemented the recommendation. The DBH has incorporated goals, process measures, and/or outcomes into FY 14-15 Mental Health Services Act (MHSA) contracts processed after the February 2014 Training referenced in the six-month response. Scope of Work goals and objectives correspond to the components of the County's MHSA Plan (County Plan) and are designed to meet the intended program outcomes. This business process has been implemented and will continue for all MHSA contracts.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

In September 2013, the County of San Bernardino Department of Behavioral Health (DBH) established a System-wide Evaluation and Outcomes Committee, with the objective to develop a system-wide framework for program goals and outcomes. Membership includes a variety of DBH and community-based organizations. The Committee has developed a matrix of goals and associated key outcomes. The goals are derived from Mental Health Services Act (MHSA) legislation, regulations, and Department of Health Care Services (DHCS) Information Notices; and key outcomes are operationalized ways of measuring progress toward the goals at regular intervals.

Based on Committee recommendations, DBH has provided training on how goals and outcomes are to be integrated into all new and renewed contracts for upcoming contract periods. To expedite effective integration of the new goal and outcomes language into contracts, the Committee has decided to update contracts, as necessary, with this new language as soon as is feasible.

In addition to these internal efforts, San Bernardino County has collaborated with the three counties involved in the State Audit to discuss the best ways to approach goals and outcomes. As a collaborative team, the four counties have developed an additional framework very similar to our internal model that provides what all agree are necessary long-term elements for strong goals and outcomes. With steady progress, this framework provides an effective approach for establishing goals and guiding outcome efforts.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

San Bernadrino County, Department of Behavioral Health is currently reviewing plan goals in Mental Health Services Act (MHSA) funded contracts in collaboration with the three other counties audited, Los Angeles, Sacramento, and Santa Clara. The four counties participated in teleconferences on September 30, and October 8, 2013 in which technical assistance was provided regarding MHSA program goal development. Since the release of the audit report, San Bernardino County has been reviewing our existing MHSA plans and related contract language to ensure alignment of program goal statements with MHSA contract language. Continued review will occur during the Fiscal Year 2013/2014 MHSA Integrated Plan planning process, with any necessary contract or plan changes being implemented in Fiscal Year 2014/2015.

California State Auditor's Assessment of 60-Day Status: Partially Implemented

Recommendation for Legislative Action

To ensure that Health Care Services can withhold MHSA funds from counties that fail to comply with MHSA requirements, the Legislature should enact legislation that clarifies Health Care Services' statutory authority to direct the State Controller's Office to withhold such funds from a noncompliant county.

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislation has not been introduced to address this recommendation.

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

All Recommendations in 2012-122

Agency responses received are posted verbatim.