Report 2020-611 Recommendation 5 Responses

Report 2020-611: California Department of Housing and Community Development: It Failed to Expedite Access to Federal Funding to Address the Impact of the COVID‑19 Pandemic on California's Homeless Population (Release Date: August 2021)

Recommendation #5 To: Housing and Community Development, Department of

To ensure that it has the data necessary to measure the effect the ESG-CV program has in addressing homelessness, the department should immediately develop and implement a plan to collect outcome information either independently or through HDIS. Also, by March 2022, the department should begin reporting annually the outcome information it collects so that it can demonstrate the effectiveness of its programs and so that decision makers can use the reported data to inform budget and policy decisions.

The HCD response does not have any additional updates from the last response on 03/21/2023 since the annual data is still not yet available. HCD will continue to work with BCSH's HDIS team to refine the reporting.

California State Auditor's Assessment of Status: Pending

The HCD response does not have any additional updates from the last response on 11/18/2022 since the annual data is still not yet available. HCD will continue to work with BCSH's HDIS team to refine the reporting.

California State Auditor's Assessment of Status: Pending

CSA has requested annual data and that is not yet available. HCD continues to work with BCSH's HDIS team and expects to have annual outcome data available in late 2022 or early 2023.

California State Auditor's Assessment of Status: Partially Implemented

We continue to assess the status of department's implementation as partial because it has taken steps to receive data from the State's Homeless Data Integration System in order to measure the performance of the ESG-CV program, but it has not yet begun to report information annually to demonstrate the effectiveness of its programs.

In previous responses, HCD provided the plan to collect outcome information and detailed the steps HCD has taken to develop both outcome measures and the logic to obtain the outcome information. The previous responses also detailed the steps taken to date to fully implement the use of HDIS for outcome reporting. Data is being pulled quarterly by BCSH HDIS staff and runs three months in arrears. The last report was from the beginning of the program through March 2022. After pulling the data, the BCSH HDIS team goes through a validation process to ensure the information is reported accurately. The March report reflected the total universe of ESG-CV funding, not just HCD's ESG-CV funding. The next report (the data will be through June 2022 and the report would likely be available in October) will have both the correct logic and the correct universe. HCD's contractor, ICF, has provided two pre-outcome measurement trainings to all California HMIS Lead Organizations to ensure that ESG-CV are set up correctly to allow accurate reporting of outcomes obtained with ESG-CV funding. An additional training is planned for the week of August 15th. HCD reports the ESG-CV outcome data requested by HUD on a quarterly basis through the HUD Sage reporting system. Each year, HCD prepares the Consolidated Annual Performance and Evaluation Report (CAPER) to report the outcomes for all HUD's annual grants to HCD. Information for ESG and ESG-CV is provided via an attachment of the Sage report to the CAPER. This reporting is for HUD's outcomes which are different from the outcome measures HCD established for ESG-CV. There is no other required reporting for ESg or ESG-CV however HCD may elect to use some of this data in the future in broader HCD reporting.

California State Auditor's Assessment of Status: Partially Implemented

Although the department has taken steps to receive data from the State's Homeless Data Integration System in order to measure the performance of the ESG-CV program, it has not fully implemented the recommendation, because it has not yet begun to report information annually to demonstrate the effectiveness of its programs. Further, the department states in its response that it prepares the Consolidated Annual Performance and Evaluation Report (CAPER) to report the outcomes for all HUD's annual grants to the department. However, as we stated in our report, the department's outcome measures focus only on output information, such as the types of services the CoCs have provided and the number of people they have served. We continue to stand by our recommendation that the department should begin reporting annually the outcome information it collects so that it can demonstrate the effectiveness of its programs and so that decision makers can use the reported data to inform budget and policy decisions.

With the completion of Phase I in February, HCD moved to Phase II in March and April. HCD worked with local HMIS leadership to assure that all HCD funded projects had the correct contract number entered data element 2.06 2 "Grant Identifier". The combination of ESG-CV (Project Type = 47) and Grant Identifier (2.06 2) provides HDIS leadership sufficient data to create the universe of projects specific to the HCD funded ESG-CV program. During the second HDIS data pull, HCD and HDIS leadership will adjust for any changes that arose from the ESG-CV outcome data measures logic from Phase I and the completion of the data universe data quality efforts that assure data element 2.06 2 "Grant Identifier" contains HCD contract numbers, thereby assuring the correct ESG-CV universe. HCD will also collect the HCD-funded ESG-CV project ids and organization ids from each subrecipient. This Phase will allow HCD to perform the first analysis of the ESG-CV data Following the successful completion of Phases I and II, the ESG-CV outcome data measures logic will be applied to the annual ESG program, and the template(s) used for ESG-CV universe identification will be turned into a job aid for annual ESG leadership that allows them to tell HDIS leadership on a quarterly basis what the changing universe of programs are that are funded and must be represented in the outcome data.

California State Auditor's Assessment of Status: Partially Implemented

Although the department has taken steps to receive data from the State's Homeless Data Integration System in order to measure the performance of the ESG-CV program, it has not fully implemented the recommendation, because it has not yet begun to report information annually to demonstrate the effectiveness of its programs.

Please see full implementation sent directly to CSA on 02/24/2022

In partnership with the Homeless Coordinating Financing Council leadership who manage the Homeless Data Integration System (HDIS), HCD developed outcome data to measure the performance of the ESG-CV program. The ESG-CV outcome measures will provide HCD the tools necessary to measure the performance of the ESG-CV program, and HCD will modify the logic below for annual ESG. This outcome performance data will be used to prioritize funding for those projects that demonstrate higher levels of performance, while deploying HCD staff to help increase the capacity of those projects that demonstrate lower levels of performance. See the attached document "ESG-CV Outcome Data Specifications."

The process to obtain outcomes measures will be completed in three phases:

Phase I: (currently in progress): Homeless Data Integration System (HDIS) leadership will pull data from HDIS in February 2022 that covers the period of ESG-CV program onset to 9.30.2021.

Phase II (will begin February 2022): HCD will work with local Homeless Management Information System (HMIS) leadership to assure that all HCD funded projects have the correct contract number.

Phase III: following the successful completion of Phase I and II the ESG-CV outcome data measures logic will be applied to the annual ESG program, and the template(s) used for ESG-CV universe identification will be turned into a job aid for annual ESG leadership.

HCD will request quarterly data pulls from HDIS Leadership through the end of the ESG-CV program and on-going for the annual ESG program to demonstrate the effectiveness of the programs and has approved ICF to provide a series of pre-outcome measure trainings with all CA HMIS Lead Organizations to ensure that ESG-CV projects are set up correctly, using HMIS as required by HUD and HCD, and reporting quality data to HDIS.

California State Auditor's Assessment of Status: Partially Implemented

In early March 2022, the department received data from the State's Homeless Data Integration System and demonstrated that it is moving forward with its process to measure the performance of the ESG-CV program. However, the department has not fully implemented the recommendation because it has not yet begun to report information annually to demonstrate the effectiveness of its programs.

HCD is working with BCSH to develop outcome data to measure the performance of the ESG-CV program. Should this ESG-CV outcome measures yield HCD with the tools necessary to measure the performance of the ESG-CV program, HCD will modify the logic below for annual ESG. At the discretion of HCD, ICF will proceed with a series of pre-outcome measure trainings with all CA HMIS Lead Organizations to ensure that ESG-CV projects are set up correctly, using HMIS as required by HUD and HCD, and reporting quality data to HDIS. This is the proposed plan; however, further work with BCSH HDIS staff will be necessary to determine what outcome data will be relevant to HCD's management of the ESG-CV Program.

After working with BCSH HDIS staff and acquiring outcome data, HCD will implement an annual process to evaluate the effectiveness of the ESG-CV program and ultimately the ESG annul program. This outcome performance data will be used to prioritize funding for those projects that demonstrate higher levels of performance, while deploying HCD staff to help increase the capacity of those projects that demonstrate lower levels of performance.

- See HDIS - ESG-CV Outcome Measures

California State Auditor's Assessment of Status: Partially Implemented

The department has developed a plan to collect outcome information and measure the performance of the ESG-CV program using the State's Homeless Data Integration System. To measure program performance, the department has identified seven outcome measures, such as the percentage of people who remained in permanent housing for a designated length of time and the percentage of people who returned to homelessness from permanent housing within a specified period of time. However, the department did not provide documentation to substantiate that it has begun collecting data to evaluate the seven outcome measures in its plan. Additionally, it has not yet begun to report information annually to demonstrate the effectiveness of its programs.

All Recommendations in 2020-611

Agency responses received are posted verbatim.