Report 2019-116 Recommendation 11 Responses

Report 2019-116: Juvenile Justice Crime Prevention Act: Weak Oversight Has Hindered Its Meaningful Implementation (Release Date: May 2020)

Recommendation #11 To: Los Angeles County Probation Department

To adequately assess the effectiveness of its programs at reducing juvenile crime and delinquency, Los Angeles should collect data on all participants in each JJCPA program and for each service it provides.

Annual Follow-Up Agency Response From September 2023

As mentioned in our prior Recommendation response follow-up (10/8/2021, Fall 2022), the County of Los Angeles, in collaboration with our JJCPA evaluator, has developed a general program evaluation design, including a JJCPA evaluation framework/metrics that also included finalizing data collection requirements for analyzing/reporting on justice outcomes, indices of service utilization, sign-in sheet templates to facilitate data collection template for data submission with youth identifiers and status in programs, data requirements for inclusion in agreements with applicable governmental agencies/utilization in solicitation processes and identified instances where the program design suggests that other outcome measurement approaches may be more appropriate. Further, the evaluator is the final stages of producing a toolkit that agencies can utilize to support the collection of evaluation data. Moving forward, Probation will continue to collect data from programs serving Probation-involved youth and will work with another County department to track the data for at-promise youth. The JJCPA evaluator will continue to provide additional technical assistance for any challenges/concerns, such as providing guidance on how to build evaluation data collection into existing program procedures and how to select additional data elements specific to the expected outcomes of each program. As more quantitative data becomes available from funded programs, our JJCPA evaluator will analyze data on youth served and youth outcomes, including data on any program-specific outcomes and juvenile justice involvement as available. Probation anticipates collection of specific data to be completed (and on-going) by the next annual response due date in order to adequately assess the effectiveness of participant programs at reducing crime and delinquency.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From December 2022

As mentioned in our response (10/8/2021), the County of Los Angeles, in collaboration with our JJCPA evaluator, has developed a general program evaluation design, including a JJCPA evaluation framework/metrics that included finalizing data collection requirements for analyzing/reporting on justice outcomes, indices of service utilization, sign-in sheet templates (Probation and at-promise youth programs) to facilitate data collection, template for data submission with youth identifiers and status in programs, data requirements for inclusion in agreements with applicable governmental agencies/utilization in solicitation processes and identified instances where the program design suggests that other outcome measurement approaches may be more appropriate. Through data focused meetings with multiple funded programs, they reviewed expectations for collection of youth identifiers and provided technical assistance for any challenges/concerns. informed by evaluability assessments they conducted on existing programs which serve as an additional opportunity to identify potential data collection challenges and determine whether there are additional data elements specific to the expected outcomes of each program that programs should consider collecting for evaluation purposes. As part of this process, our JJCPA evaluator conducted targeted literature searches to identify outcomes relevant to programs like those funded through JJCPA and to make recommendations on specific measures that programs could consider utilizing. The collection of data for specific full program evaluation has been finalized for the first phase of programs under evaluation. Our JJCPA evaluator is targeting the analysis of quantitative data on youth served and potential outcome data for preliminary reporting once the County finalizes the data collection for the remaining programs, particularly those which serve at-promise youth.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2021

As a follow-up to our 1-Year response, the County of Los Angeles, in collaboration with our JJCPA evaluator, has developed a general program evaluation design, including a JJCPA evaluation framework and metrics that included finalizing: data collection requirements for analyzing/reporting on justice outcomes, indices of service utilization (e.g., number of youth served across programs and youth demographics), templates for sign-in sheets for Probation/at-promise youth programs to facilitate data collection, a template for submission of data with youth identifiers, status in programs and data requirements for inclusion in applicable governmental agencies' agreements/utilization in solicitation processes. Our evaluator also identified instances where the program design suggests other outcome measurement approaches may be more appropriate, held data focused meetings with multiple funded programs to review the expectations for collection of youth identifiers and provide technical assistance for any challenges or concerns. The evaluator is currently conducting evaluability assessments on existing programs, which serve as an additional opportunity to identify potential data collection challenges and determine whether there are additional data elements specific to the expected outcomes of each program that programs should consider collecting for evaluation purposes. As part of this process, our evaluator also conducted targeted literature searches to identify outcomes relevant to programs like those funded through JJCPA and to make recommendations on specific measures that programs could consider utilizing. Additionally, the collection of data for specific full program evaluation is in process of finalization. Our evaluator is targeting the analysis of quantitative data on youth served and potential outcome data for preliminary reporting prior to the next 1-Year audit response due during the Fall of 2022.

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

As a follow-up to our 6-Month response, the County of Los Angeles is working with our JJCPA Evaluator to document confirmation of data collected on "all participants in each JJCPA program and service to adequately assess the effectiveness of those programs at reducing juvenile crime and delinquency" for Base and one-time Growth funded programs. This process will be completed in lieu of direct State Auditor's Office follow-up, as discussed with their Office after the 6-Month response was submitted. Additionally, there were some unanticipated delays in completing this process due to COVID-19. It is anticipated that this process will be completed prior to the next 1 Year audit response due May 12, 2022.

California State Auditor's Assessment of 1-Year Status: Pending

Los Angeles continues to work with an evaluator to, among other objectives, advise the county about data collection. Consequently, our assessment of the county's implementation of this recommendation is pending the evaluator's report on data collection and Los Angeles's ability to demonstrate that it is collecting data on all participants in each JJCPA program and for each service it provides. Although Los Angeles indicates it will complete this process in lieu of our office's follow-up, we will continue to review the status of this recommendation as part of our regular follow-up process.

6-Month Agency Response

As a follow-up to our 60-Day response, the County of Los Angeles has available data for JJCPA (Base) funded programs for Fiscal Year (FY) 2019-2020. Additionally, the JJCPA Evaluator is working with the County to evaluate identified programs during FY 2020-2021, with a plan to continue this process through the next several FYs.

California State Auditor's Assessment of 6-Month Status: Pending

As referenced in its response, Los Angeles has contracted with an evaluator to, among other objectives, advise the county about data collection. Consequently, our assessment of the county's implementation of this recommendation is pending the evaluator's report on data collection and Los Angeles's ability to demonstrate that it is collecting data on all participants in each JJCPA program and for each service it provides.

60-Day Agency Response

The County of Los Angeles is in the process of finalizing data collection on JJCPA program participants for Fiscal Year 2019-2020 (July 1, 2019 - June 30, 2020); this process is targeted to be completed by October 2020 and updated information will be provided in the 6-month audit response due on November 12, 2020.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2019-116

Agency responses received are posted verbatim.