Report 2012-122 Recommendation 6 Responses

Report 2012-122: Mental Health Services Act: The State's Oversight Has Provided Little Assurance of the Act's Effectiveness, and Some Counties Can Improve Measurement of Their Program Performance (Release Date: August 2013)

Recommendation #6 To: Health Care Services, Department of

To ensure that Health Care Services and other state entities can evaluate MHSA programs and assist the Accountability Commission in its efforts, Health Care Services should collect complete and relevant MHSA data from the counties.

Annual Follow-Up Agency Response From October 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve CSI and DCR data quality. Counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement, MHDA, has produced quarterly reports which contain the client counts, service counts, and percent error of data submissions to support monitoring and assistance for counties in improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created EPLD templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county data quality data. For the remainder of FY 2019-20, MHDA will train DHCS on use of the EPLD template, so county assistance can fully transition to DHCS staff. MHDA will continue to work closely with DHCS on CSI data monitoring and communicating with counties as needed. In FY 2020-21, MHDA and DHCS will continue to collaborate to transition CSI/DCR county technical assistance and training functions to DHCS staff. CSI corrective action plans will be created for counties and MHDA will conduct CSI data submission webinars for counties and provide technical assistance to vendors regarding business rules. For DCR, MHDA will provide additional training materials and quarterly trainings to counties in addition to data quality reports.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2019

DHCS continues to work closely with counties, providing in-depth technical assistance to improve the quality of the CSI and DCR data. Specifically, counties were issued CSI Data Quality Improvement Issue reports which identified county-specific data entry issues and the steps necessary to correct the errors identified. Forty-three counties submitted Plans of Correction and DHCS is monitoring the timelines for correction. The contractor for CSI/DCR Data Quality Improvement, Mental Health Data Alliance (MHDA), has produced quarterly reports which contain the client counts, service counts, plus percent error of data submissions to further enable continued monitoring and assistance for counties and their progress towards improving data submission. In addition to this, MHDA has updated the DCR User Manual and provided a technical assistance webinar for counties. MHDA also created Enhanced Provide-Level Data (EPLD) templates for DCR data in the Structured Query Language Environment at DHCS so that DHCS staff can better monitor county progress and compliance with submitting quality data. For the remainder of FY 2019-20, MHDA will provide training to DHCS regarding the use of the EPLD template so that assistance to counties can fully transition to DHCS staff. MHDA will also continue to work closely with DHCS in monitoring the improvement of CSI data and communicating with counties as needed.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Health Care Services is taking steps to collect complete and relevant MHSA data from the counties including Plans of Correction to address and correct data reporting errors. Health care Services is also working with a vendor to fully assume the vendor's role in monitoring county data.

Annual Follow-Up Agency Response From November 2018

DHCS entered into a vendor contract to improve data quality and completeness in the CSI and the MHSA DCR. The vendor created reports of findings (e.g., data entry errors, anomalies, missing data) and delineated the need for improved infrastructure and maintenance of data quality. The vendor has also provided customized technical

assistance (e.g., weekly webinars, monthly site visits) for counties and EHR vendors to improve data quality. DHCS is already seeing improvements in data quality. For years, client counts in CSI had been declining. Since the vendor began providing reports and technical assistance to the counties, the client counts have increased statewide by 7.6 percent for FY 2016-17 data. Since data quality is a continuous process, DHCS has extended the vendor's contract to FY 2018-19, and is negotiating a contract for FY 2019-20. Ultimately, DHCS plans to transition vendor-provided technical assistance and system training to DHCS staff within the contract timeframe for continued county support. This recommendation will be fully implemented in June 2020.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From August 2017

DHCS contracted with a vendor to perform data clean up on the data included in the DHCS Client Services Information System and MHSA Data Collection and Reporting System. The purpose of the project is to provide DHCS, the MHSOAC, and the counties with accurate and complete data that may be used to track performance outcomes for Community Services and Supports, Full Service Partnership clients. This project is scheduled for completion over the next 12-18 months. With DHCS maintaining data quality thereafter. Counties have already received their comprehensive data error reports, and technical assistance calls to support the interpretation and use of these reports are currently underway. The data clean up project is to be completed by Fall 2018. Upon completion, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Partially Implemented

Annual Follow-Up Agency Response From September 2016

DHCS is utilizing a contractor to help improve data reporting timeliness and data quality in the Client and Service Information system and the Data Collection and Reporting system.

Work on this project is will begin in FY 2016-17.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Annual Follow-Up Agency Response From October 2015

All of the data that DHCS collects is relevant data. DHCS is working to improve the timeliness of the data that is reported and the completeness of the data that is reported. DHCS has implemented processes that have improved the timeliness of data reporting in both the Client and Services Information (CSI) data system and the Data Collection and Reporting (DCR) data system. Counties are now reporting data timely.

- Client and Services Information (CSI) Data: According to the DHCS September 2015 county submission status report, 50 counties are current in submitting. County submissions average 86% current with data submission.

- Data Collection Reporting (DCR): According to the DHCS September 2015 county submission status report, 57 counties are current in submitting DCR full service partnership data. County submissions average 97%.

DHCS is starting to focus on improving the completeness of the data submitted. Once DHCS is completed the work necessary to improve the completeness of data, this recommendation will be fully implemented.

DHCS is currently working on relevancy in its data modernization project. Once the data modernization project is done, this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

1-Year Agency Response

DHCS continues to maintain monthly contact with county staff for obtaining status updates and providing technical assistance.

Client and Services Information (CSI):

According to the DHCS June 2014 county submission status report, 46 counties are current in submitting CSI data. The number of counties submitting late went from 9 counties in January 2014 to 12 counties in June 2014. Late submissions went from an average of 7 months delayed in January 2014, to an average of 3 months delayed by June 2014.

Data Collection Reporting (DCR):

According to the DHCS June 2014 county submission status report, 56 counties are current in submitting DCR full service partnership data. The number of counties submitting late decreased from 8 counties in January 2014 to 3 counties in June 2014. The number of counties that never submitted data decreased from 5 counties in January 2014 to 2 counties in June 2014. Late submissions went from 3 counties being an average of 8 months late in January 2014 to 1 county being 21 months late in June 2014.

California State Auditor's Assessment of 1-Year Status: Partially Implemented

6-Month Agency Response

For all counties late in reporting data, DHCS maintains monthly contact with county staff to obtain an update on their reporting status and timeline for becoming current with data reporting, and to offer any technical assistance. Submitted reports are attached to this document.

Client and Services Information (CSI): According to the DHCS January 2014 county submission status report, 49 counties are current in submitting CSI data. The number of counties submitting late decreased from 14 counties in September 2013 to 9 counties in January 2014. Late submissions went from an average of 10 months delayed in September 2013 to an average of 7 months delayed by January 2014.

Data Collection Reporting (DCR): According to the DHCS January 2014 county submission status report, 51 counties are current in submitting DCR full service partnership data. The number of counties submitting late decreased from 9 counties in September 2013 to 8 counties in January 2014. Late submissions went from an average of 15 months late in September 2013 to an average to 8 months late by January 2014.

California State Auditor's Assessment of 6-Month Status: Partially Implemented

60-Day Agency Response

DHCS continues to review county data submission for completeness and accuracy. According to the DHCS September 2013, county submission status reports, 44 counties are current in submitting Client and Services Information (CSI) data and 14 are not current in reporting. Regarding the Data Collection and Reporting (DCR) full service partnership data, 49 counties are current in submitting DCR data and 9 are not current.

To assist all counties in becoming fully compliant with their required reporting, DHCS established and operates a helpdesk for county data reporting questions; provides technical assistance to county and county vendor information technology staff; posts monthly county submission status reports on the DHCS website; and coordinates with CMHDA and the Accountability Commission for any needed system improvements and updates to data reporting requirements. In September 2013, DHCS began contacting the County Mental Health Director in each county without current data to request the county's reporting status and timeline for becoming current with data reporting.

DHCS will begin an in-depth assessment of CSI and DCR data completeness and relevancy for outcomes reporting purposes in the winter of 2013.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2012-122

Agency responses received are posted verbatim.