2016-041 - Status of Recommendations - Table 2

2. To demonstrate its willingness to fairly evaluate regional expenditures, Education should allow San Joaquin to reimburse its general fund for the vehicle purchase Education incorrectly disallowed.

13. To determine if the statewide migrant education program is effective, Education should finalize its current evaluation of the program and begin developing the capacity to produce a more robust annual evaluation of the program.

14. To address a lack of detailed migrant program service and outcome data, Education should either expand the capabilities of its existing statewide databases or implement additional systems that would allow regions to capture more detailed data about migrant students.

19. To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should use data from the kids survey and reported suspensions and expulsions to evaluate the levels of discrimination, harassment, intimidation, and bullying students encounter and to determine the effectiveness of its own and the LEAs' efforts, and report the results to the Legislature by August 1, 2014.

21. To provide stronger leadership with respect to school safety and nondiscrimination laws, Education, with direction from the superintendent of public instruction, should within the next six months and annually thereafter, update and replace the resources on its website to provide more relevant information on best practices, such as preventing and responding to incidents related to a protected characteristic or that occur through cyberbullying, the U.S. DOE report on state bullying legislation, and best practices in other states, such as the Massachusetts law on LEA staff training requirements.

1. To ensure that colleges receive consistent and fair treatment and are able to address deficiencies, the chancellor's office should work with the community colleges and request clearer guidance from the commission regarding what actions would allow for the full two-year period in which to remediate concerns and what actions would constitute good cause for extending the time an institution has to address deficiencies beyond two years. In doing so, the chancellor's office should also encourage the commission to specify in its policies those scenarios under which it would exercise the good cause exception so that institutions would have a better understanding of when they might reasonably expect additional time to address deficiencies.

2. To ensure that community colleges and the public are fully informed regarding the accreditation process, the chancellor's office should assist community colleges in communicating their concerns to the commission regarding its transparency and in developing proposals for improving the commission's transparency policies and practices. The chancellor's office should also encourage the commission to publish policies describing the role of its staff in the commission's decision-making processes.

3. To make certain that institutions receive fair treatment in appealing decisions that terminate their accreditation, the chancellor's office should work with the community colleges to advocate that the commission change certain aspects of its appeal process. Specifically, in keeping with the spirit of accreditation, when institutions have taken steps to correct deficiencies that led to the decision to terminate accreditation, the institutions should be allowed to have information on those corrections heard as evidence in their appeal. Further, the commission president's involvement in selecting the appeal panel's counsel should be revisited.

4. To strengthen institutions' understanding of what they must do to comply with standards, and to provide them with the opportunity to address certain issues that could jeopardize their compliance, the chancellor's office, in collaboration with the community colleges, should encourage the commission to develop formal opportunities for institutions to communicate with and receive feedback from the commission on institutional self-studies and other reports before a formal evaluation takes place. In doing so, the chancellor's office should consider the practices of other regional accreditors and identify those that would best meet the needs of California's community colleges.

5. Community colleges, as members of the commission, should communicate their concerns about and ideas for improvement of training on the accreditation process to the commission. To provide assurance to colleges that they may suggest this information freely, the chancellor's office should coordinate communication between the commission and the colleges. Further, in order to build collegial relationships, engage new people in the accreditation process, and extend additional training to those already involved in accreditation, the chancellor's office should encourage the commission to develop an annual conference focused on accreditation and oversight.

7. To allow colleges flexibility in choosing an accreditor, the chancellor's office should identify other accreditors who are able to accredit California community colleges or who would be willing to change their scopes to do so.

8. To allow colleges flexibility in choosing an accreditor, the chancellor's office should assess the potential costs, risks, and feasibility of creating a new independent accreditor.

20. To ensure that its respective institutions comply with the Clery Act, the Community Colleges Chancellor's Office should develop written policies and procedures to provide guidance to its institutions on how to report accurate Clery Act crime statistics and ensure that all required disclosures are included in its respective institutions' annual security reports. The Community Colleges Chancellor's Office should then annually revisit the written policies and procedures to ensure that they are up to date.

4. Within six months of the date the Legislature clarifies its intent regarding Education Code Section 89708, the Chancellor's Office should develop and issue final guidance to campuses regarding supplanting, including identifying appropriate oversight mechanisms for ensuring campuses' compliance with this law.

19. To ensure that its respective institutions comply with the Clery Act, the CSU Office of the Chancellor should develop written policies and procedures to provide guidance to its institutions on how to report accurate Clery Act crime statistics and ensure that all required disclosures are included in its respective institutions' annual security reports. The CSU Office of the Chancellor should then annually revisit the written policies and procedures to ensure that they are up to date.

1. To address the variations in per student funding of its campuses, the university should complete its reexamination of the base budgets to the campuses and implement appropriate changes to its budget process. As part of its reexamination of the base budget, it should:
X Identify the amount of general funds and tuition budget revenues that each campus receives for specific types of students (such as undergraduate, graduate, and health sciences) and explain any differences in the amount provided per student among the campuses.
X Consider factors such as specific research and public service programs at each campus, the higher level of funding provided to health sciences students, historical funding methods that favored graduate students, historical and anticipated future variations in enrollment growth funding, and any other factors applied consistently across campuses.
X After accounting for the factors mentioned above, address any remaining variations in campus funding over a specified period of time.
X Make the results of its reexamination and any related implementation plan available to stakeholders, including the general public.

8. To ensure that campuses do not inappropriately use revenues generated from student fees imposed by referenda, the university should ensure that it, the regents, and the campuses do not expand the uses for such revenues beyond those stated in the referenda.

23. To address the improper acts we identified, the university should collect $1,802 from the official for the wasteful expenses he claimed for lodging and meals during his trip to England, the expenses he incurred within the vicinity of his headquarters, and the business meal expenses.

27. The university should revise policies to establish defined maximum limits for the reimbursement of domestic lodging costs and establish controls that allow for exceptions to the limits under specific circumstances only.

5. The Office of the President should direct all of the universities within the UC system to comply with the recommendations in this audit report. Also, to ensure that its universities are complying with Title IX requirements, the Office of the President should conduct routine Title IX reviews. When conducting these compliance reviews, the Office of the President should determine whether universities have implemented this report's recommendations.

49. The Office of the President should clarify in the UC policies that a complainant must have and be informed about the right to end the early resolution process at any time and request that his or her complaint be handled under the university's formal process.

51. The Office of the President should clarify in the UC policies that if university officials approve an extension to an investigative timeline, the extension should be restricted to a single extension of no more than 30 days, except in limited circumstances that are beyond the university's control.

18. To ensure that its respective institutions comply with the Clery Act, UCOP should finalize and implement its draft policy that will provide additional guidance and oversight to its institutions.

8. To help ensure that university faculty and staff do not mishandle student reports of incidents, all faculty and staff should receive training annually, consistent with their role, on their obligations in responding to and reporting incidents of sexual harassment and sexual violence.

20. All universities should provide their education on sexual harassment and sexual violence to incoming students as close as possible to when they arrive on campus but no later than the first few weeks of their first semester or quarter. Further, universities should provide periodic refresher educational programs, at least annually, to all students on campus to ensure that they are aware of how to handle and report incidents of sexual harassment and sexual violence.

58. All universities should ensure that the differences between an informal or early resolution process and a formal investigation process are clearly explained to ensure that students know what to expect from each process. Further, they should explain that students whose cases are being handled under an informal or early resolution process have the right to move to a formal process at any time.

4. UC Davis should collect all late fees that its licensees owe.

6. UC Davis should develop a risk-based audit plan to begin periodically reviewing the financial records of master licensees and licensed nurseries to ensure that they are accurately reporting all of their sales of licensed strawberry varieties and paying the university all the royalties it is entitled to. To encourage compliance, UC Davis should notify all master licensees and licensed nurseries that it will begin auditing the sales records of selected licensees.

9. To help ensure that university faculty and staff do not mishandle student reports of incidents, all faculty and staff should receive training annually, consistent with their role, on their obligations in responding to and reporting incidents of sexual harassment and sexual violence.

21. All universities should provide their education on sexual harassment and sexual violence to incoming students as close as possible to when they arrive on campus but no later than the first few weeks of their first semester or quarter. Further, universities should provide periodic refresher educational programs, at least annually, to all students on campus to ensure that they are aware of how to handle and report incidents of sexual harassment and sexual violence.

59. All universities should ensure that the differences between an informal or early resolution process and a formal investigation process are clearly explained to ensure that students know what to expect from each process. Further, they should explain that students whose cases are being handled under an informal or early resolution process have the right to move to a formal process at any time.

18. To improve its enforcement, each year Public Health should evaluate the effectiveness of its enforcement system across all types of health facilities, including those in developmental centers, prepare the required annual report, and, if called for, recommend legislation to improve the enforcement system and enhance the quality of care.

3. To protect the health, safety, and well-being of residents in long-term health care facilities, Public Health should improve its oversight of complaint processing. Specifically, by May 1, 2015, Public Health should establish a specific time frame for completing facility-related complaint investigations and ERI investigations and inform staff of the expectation that they will meet the time frame. Public Health should also require district offices to provide adequate, documented justification whenever they fail to meet this time frame.

4. To protect the health, safety, and well-being of residents in long-term health care facilities, Public Health should improve its oversight of complaint processing. Specifically, by May 1, 2015, Public Health should develop formal written policies and procedures for PCB to process complaints about certified individuals in a timely manner. These policies and procedures should include specific time frames for prioritizing and assigning complaints to investigators, for initiating investigations, and for completing the investigations. Public Health should also inform staff of the expectation that they will meet these time frames. It should require PCB to provide adequate, documented justification whenever PCB fails to meet the time frames.

7. To ensure that district offices address ERIs consistently and to ensure that they investigate ERIs in the most efficient manner, Public Health should review periodically a sample of the priorities that district offices assign to ERIs to ensure compliance with best practices.

8. To protect the residents in long-term health care facilities from potential harm, Public Health should ensure that its district offices have adequate staffing levels for its licensing and certification responsibilities, including staffing levels that allow prompt investigations of complaints. Specifically, Public Health should continue working with CalHR to complete the reclassification of district offices' investigator supervisor and manager positions and then quickly fill the vacant positions at district offices.

12. To ensure that its district offices properly investigate complaints and ERIs, Public Health should make certain that all district offices follow procedures requiring supervisory review and approval of complaint and ERI investigations. If the district offices do not have a sufficient number of supervisors to review investigations they did not conduct, Public Health should arrange to assist the districts until such time that they do have a sufficient number of supervisors.

13. To make certain that its district offices comply with federal requirements regarding corrective action plans, Public Health should establish a process for its headquarters or regional management to inspect district office records periodically to confirm that they are obtaining corrective action plans according to the required time frame and verifying that facilities have performed the corrective actions described in the plans when required.

14. To ensure that it has closed complaints and ERIs appropriately, Public Health should take steps by April 2015 to verify that complaints that its field operations branch closed administratively were closed appropriately. For example, it could request the district offices to verify that the closures were appropriate.

2. To increase its efforts to prevent and control diabetes, Public Health should develop a process for identifying and applying for federal funding opportunities, including routinely and proactively searching for grants. In addition, Public Health should seek funding for a grants specialist position to identify and apply for federal and other grants.

2. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should inspect all in-state and out-of-state labs it has licensed every two years.

6. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should work with Public Health's budget section and other appropriate parties in developing a process to assess the budget act annually and to adjust its fees accordingly. The process should include its management's review and approval of fee adjustments before it posts those fees publicly.

8. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should address staffing issues by preparing and resubmitting to Public Health a recruitment and retention proposal, developing a succession plan, and taking necessary steps to implement its planned reorganization.

9. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should ensure that its information technology data systems have necessary safeguards, contain accurate and complete data, and support its program needs.

10. To ensure it can provide effective oversight of labs as state law requires, Laboratory Services should update and develop its regulations as necessary to ensure consistency with existing state law.

2. To encourage more effective communication from county CWS agencies regarding its licensees, Social Services should specify in regulations what types of situations or allegations the agencies should forward to its licensing division.

3. To ensure that rates paid to foster family agencies are appropriate, Social Services should analyze the rates and provide reasonable support for each component, especially the 40 percent administrative fee it currently pays these agencies.

4. Social Services should create and monitor compliance with clear requirements specifying that children placed with foster family agencies must have elevated treatment needs that would require a group home placement if not for the existence of these agencies' programs. Specifically, Social Services should revise its regulations so licensed foster homes have higher priority than foster family agencies for children that do not have identified treatment needs.

5. Social Services should require county CWS agencies to file in CWS/CMS a detailed justification for any child placed with a foster family agency.

6. Social Services should create a mechanism by which it can efficiently check for compliance with the needs-justification requirement.

8. To determine whether the hold harmless provision has been effective in reducing caseloads and whether it should be revised or rescinded, Social Services should refine and use CWS/CMS to calculate and report county CWS caseloads.

9. To encourage county CWS agencies to conduct formal internal death reviews, Social Services should revise its annual report on child deaths resulting from abuse or neglect to provide information on whether county CWS agencies conducted such a review of child deaths with prior CWS history. To obtain this information, Social Services should revise its regulations to require all county CWS agencies to not only report child deaths resulting from abuse or neglect but to also require a subsequent report indicating whether an internal child death review was completed.

11. To provide more useful information in its annual report, Social Services should provide child death information broken out by county, not just statewide totals. Further, Social Services should provide more analysis, such as comparing child death information over multiple years and presenting each county's child deaths as a percentage of its total child population.

40. To promote continued improvement in the CWS system, Social Services should encourage each county CWS agency to designate personnel to update regularly their policies and procedures, to include a detailed description of the need for ongoing supervisory reviews of key aspects of their respective service processes and incorporate that description into their policies and procedures, and to designate personnel to perform regular quality assurance reviews.

41. To promote continued improvement in the CWS system, Social Services should ask each county CWS agency to report to Social Services on the status of these efforts within 60 days, six months, and one year from the publication of this audit report.

4. To ensure that all counties consistently gauge the cost-effectiveness of their early fraud detection activities and ongoing investigation efforts for the CalWORKs and CalFresh programs, Social Services should develop a formula to regularly perform a cost-effectiveness analysis using information that the counties currently submit. Specifically, this formula should measure the savings that a county achieves for each dollar spent on antifraud efforts.

5. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should, using the results from the recommended cost-effectiveness analysis, determine why some counties' efforts to combat welfare fraud are more cost-effective than others.

6. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should seek to replicate the most cost-effective practices among all counties. Social Services should work with its legal counsel to determine whether to withhold information about these practices from public disclosure.

7. Social Services should track counties' prosecution thresholds for welfare fraud cases and determine whether they affect counties' decisions to investigate potential fraud, with a focus on determining best practices and cost-effective thresholds. If Social Services' analysis determines that varying prosecution thresholds do affect counties' decisions, it should then work with counties to implement the consistent use of these cost-effective prosecution thresholds.

8. Social Services should continue its efforts to ensure that counties follow state regulations regarding the use of the administrative disqualification hearings process until all counties have adopted the process.

9. To make certain that counties receive the greatest benefit from the resources they spend on antifraud efforts related to CalWORKs and CalFresh cases, Social Services should address and promptly act on the four remaining recommendations that its steering committee provided in 2008.

11. To ensure that counties are consistently following up on all match lists, Social Services should remind counties of their responsibility under state regulations to follow up diligently on all match lists. Further, it should work with counties to determine why poor follow-up exists and address those reasons.

12. To make counties' review of match lists more efficient, Social Services should revive its efforts to work with the state and federal agencies that prepare the match lists to address the counties' concerns about match list formats, content, and criteria.

13. To ensure the accuracy of the overpayments that counties collect and report for the CalFresh program, Social Services should create a process to verify on a rotational basis the counties' overpayment collection reports.

16. To ensure the accuracy and consistency of the information on welfare fraud activities that counties report and that Social Services subsequently reports to the federal government, the Legislature, and internal users, Social Services should incorporate the upcoming federal changes to the revision of its instructions for completing the counties' investigation activity reports. In the interim, Social Services should issue clarifications for the most common errors Social Services observes counties make in reporting their investigation activities.

1. To ensure that all address matches of registered sex offenders who potentially reside or work at a licensed facility or foster home are reviewed, Social Services should improve its current mechanism to track and monitor the outcome of each address match it identifies. This tracking mechanism should allow Social Services to actively reconcile the number of address matches identified through its address comparison process with the number of completed reviews to ensure that it appropriately reviewed each match. Further, this mechanism should allow Social Services to actively monitor and report on any overdue investigations.

3. To ensure that counties' use of foster family agency placements is justified, Social Services should take action to implement the recommendation we previously made in our 2011 audit. Specifically, Social Services should continue working to revise its rates paid to foster family agencies and to ensure that it has reasonable support to justify each rate component, especially the administrative fee it currently pays these agencies.

4. To ensure that counties' use of foster family agency placements is justified, Social Services should take action to implement the recommendation we previously made in our 2011 audit. Specifically, Social Services should require counties to give licensed foster homes a higher priority than foster family agencies for children that do not have identified treatment needs.

5. To ensure that counties' use of foster family agency placements is justified, Social Services should take action to implement the recommendation we previously made in our 2011 audit. Specifically, Social Services should require counties to prepare a detailed justification for any child placed with a foster family agency.

11. To minimize the need for overtime, the department should reassess its minimum staffing requirements, hire a sufficient number of employees to cover these requirements, and examine its employee scheduling processes.

2. To ensure timelier fee assessments, Developmental Services should hold regional centers accountable for providing the monthly placement reports and copies of information letters required by state regulations. To encourage compliance, Developmental Services should specify in its regional center contracts that noncompliant regional centers will pay financial penalties equal to the amount of revenue lost because of their inaction.

8. Developmental Services should review and update its process for collecting on delinquent accounts. This update should include a revision to the policies and procedures manual, training for field agents, and regular management review to ensure consistent adherence to the policy. As part of the update, Developmental Services should clarify when to designate an account as uncollectible.

4. Caltrans, the Energy Commission, Food and Agriculture, and Health Care Services should put in writing those policies and procedures related to intellectual property that they believe are necessary and appropriate to enable their staff to identify, manage, and protect their intellectual property.

2. To ensure that it monitors counties to the fullest extent as the MHSA specifies and that it implements best practices, Health Care Services should conduct comprehensive on-site reviews of county MHSA programs, including verifying county compliance with MHSA requirements.

3. To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should coordinate with the Accountability Commission and issue guidance or regulations, as appropriate, for Facilities programs and for other MHSA requirements, such as a prudent reserve.

5. To ensure that counties have the needed guidance to implement and evaluate their MHSA programs, Health Care Services should collaborate with the Accountability Commission to develop and issue guidance or regulations, as appropriate, to counties on how to effectively evaluate and report on the performance of their MHSA programs.

6. To ensure that Health Care Services and other state entities can evaluate MHSA programs and assist the Accountability Commission in its efforts, Health Care Services should collect complete and relevant MHSA data from the counties.

7. To ensure that Health Care Services and other state entities can evaluate MHSA programs and assist the Accountability Commission in its efforts, Health Care Services should resolve all known technical issues with the partnership and client services systems and provide adequate and expert resources to manage the systems going forward.

18. Health Care Services should develop standardized data collection guidelines or regulations, as appropriate, that will address inconsistencies in the data that counties report to the State. In developing these guidelines or regulations, Health Care Services should consult with the Accountability Commission to ensure that data collected reasonably fulfill statewide evaluation purposes.

19. To help ensure county compliance with stakeholder regulations, Health Care Services should provide technical assistance to counties on the MHSA local planning review process and ensure that its guidance to counties is clear and consistent with state regulations.

1. To ensure that the providers receive reimbursement for only valid services, Health Care Services should immediately coordinate with the appropriate counties to recover inappropriate payments to ineligible providers and for services purportedly rendered to deceased beneficiaries.

3. To ensure that the providers receive reimbursement for only valid services, Health Care Services should immediately direct its investigations division to determine whether it authorized any improper payments to program providers for deceased beneficiaries outside of our audit period. It should also determine whether it authorized such payments through its other Medi-Cal programs. Health Care Services should initiate efforts to recover such payments as appropriate.

10. To ensure that the providers receive reimbursement for only valid services, Health Care Services should immediately ensure that Los Angeles County strengthens its provider contract monitoring process, including fully implementing its RATE system to track and respond to provider deficiencies, and that it imposes appropriate responses when warranted, such as withholding payment or suspending or terminating a contract.

18. To prevent the certification of ineligible providers, Health Care Services should immediately ensure that its enrollment division conducts LEIE and EPLS database searches of program providers at least monthly.

20. To prevent the certification of ineligible providers, Health Care Services should immediately establish a mechanism to identify the number of program sites the provider applicants' medical directors work at, and ensure that the physician ratio does not exceed 1-to-3 in accordance with state law and the certification standards.

21. To prevent the certification of ineligible providers, Health Care Services should immediately identify and perform an immediate recertification of providers that signed the Compliance Agreement to ensure that these providers are currently meeting all program requirements.

24. To prevent the certification of ineligible providers, Health Care Services should immediately develop a schedule for recertifying all program providers every five years.

25. To prevent the certification of ineligible providers, Health Care Services should immediately continue its implementation of an automated provider enrollment system.

26. To prevent the certification of ineligible providers, Health Care Services should immediately complete its program recertification on or before March 24, 2016, as federal regulations require.

27. To prevent the certification of ineligible providers, Health Care Services should immediately establish a plan for eliminating its backlog of applications for new sites and services and changes to existing certifications.

33. To improve the coordination between its divisions, branches, and units and ensure that it addresses allegations of fraud in a timely manner, Health Care Services should fully implement the investigations division's recommendations shown in Appendix B. If it chooses not to implement a recommendation, it should document sufficiently the reasons for its decision.

4. To ensure that child beneficiaries throughout California can reasonably access dental services under Medi-Cal and to increase child beneficiary utilization and provider participation, Health Care Services should take the following steps for the fee-for-service delivery system by May 2015: immediately take action to resolve any declining trends identified during its monitoring efforts.

5. To help increase the number of providers participating in the program's fee-for-service delivery system, Health Care Services should improve its identification and implementation of changes that minimize or simplify administrative processes for providers. These changes should include revising its processes pertaining to dental procedures that require radiographs or photographs.

7. To ensure that the influx of beneficiaries resulting from recent changes to federal and state law is able to access Medi-Cal's dental services, Health Care Services should immediately take action to resolve any declining trends identified during its monitoring efforts.

9. To make certain that access to dental services for child beneficiaries is comparable to the access available to the general population in the same geographic areas, Health Care Services should immediately adhere to its monitoring plan.

10. To make certain that access to dental services for child beneficiaries is comparable to the access available to the general population in the same geographic areas, Health Care Services should also compare its results for measuring the percentage of child beneficiaries who had at least one dental visit in the past 12 months with the results from the three surveys conducted by other entities, as its state plan requires.

19. To ensure that it reports in the CMS-416 an accurate number of child beneficiaries who received specific types of dental services from the centers and clinics, Health Care Services should continue working on a solution to capture the details necessary to identify the specific dental services rendered.

20. To make certain that it meets the requirements of the new state law and that its performance measures are accurate, Health Care Services should establish the provider-to-beneficiary ratio statewide and by county as performance measures designed to evaluate access and availability of dental services and include this measure in its October 2015 report to the Legislature.

23. To ensure that Health Care Services and its fiscal intermediaries reimburse providers only for services rendered to eligible beneficiaries, Health Care Services should obtain Social Security's Death Master File and update monthly its beneficiary eligibility system with death information.

24. To ensure that Health Care Services and its fiscal intermediaries reimburse providers only for services rendered to eligible beneficiaries, Health Care Services should do the following: Coordinate with the appropriate fiscal intermediaries to recover inappropriate payments made for services purportedly rendered to deceased beneficiaries, if necessary.

3. To ensure that Managed Health Care reaches accurate conclusions during its quarterly assessments of the adequacy of provider networks, Health Care Services should establish by September 2015 a process to verify the accuracy of the provider network data it receives from health plans and forwards to Managed Health Care. For example, Health Care Services could verify, for a sample of physicians claimed as part of the health plans' provider networks, that health plans have current written agreements with the providers.

4. To improve the accuracy of provider directories, by December 2015 Health Care Services should revise its processes for monitoring health plans' provider directories. Specifically, Health Care Services should review how each health plan updates and verifies the accuracy of the directory. In addition, Health Care Services should identify best practices and require the plans to adopt those practices.

6. If Health Care Services finds significant errors in a health plan's provider directory, it should work with that health plan to identify reasons for the inaccuracies and require the health plan to develop processes to eliminate the inaccuracies.

7. To ensure that it can handle adequately the volume of calls from Medi-Cal beneficiaries, Health Care Services should implement an effective plan to upgrade or replace its telephone system and database to make certain that its ombudsman office can handle the volume of calls and maintain complete data to make informed management decisions.

8. To further ensure that it can handle adequately the volume of calls from Medi-Cal beneficiaries, after upgrading or replacing its systems, if Health Care Services believes that it does not have adequate staffing to address workload, it should justify its need and request additional staff.

9. To make certain that Health Care Services complies with state law requiring it to conduct annual medical audits, it should finish developing and begin adhering to its schedule for auditing all health plans in fiscal year 2015-16.

10. To ensure that Health Care Services complies with state law, it should increase its oversight of Managed Health Care to ensure that it completes the quarterly assessments required under the agreements.

1. To ensure that it provides claiming units with reasonable opportunities to address concerns with its decisions or actions, Health Care Services should, within three months, begin preparing regulations to establish and implement a formal appeals process that allows claiming units to directly appeal Health Care Services' decisions.

2. To ensure that it provides claiming units with reasonable opportunities to address concerns with its decisions or actions, Health Care Services should, within three months, inform all stakeholders, including claiming units, of the existence of this appeals process.

3. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should update its site review and desk review procedures to include the following steps:
* A risk-based approach to selecting entities for review.
* Verification that local educational consortia and local governmental agencies are adequately meeting the oversight and administrative responsibilities described in their contracts with Health Care Services.
* Verification that contracts between local educational consortia or local governmental agencies and their claiming units do not include provisions that could result in disallowed costs, such as allowing Health Care Services' participation fee to be included in the claim calculations.
* Examination of local educational consortia and local governmental agencies' records to ensure that:
-- Costs they claim for federal reimbursement are necessary and reasonable.
-- The entities are not inappropriately earning a profit based on the fees they collect from claiming units.
-- The coding performed by local educational consortia that charge claiming units a percentage of their federal reimbursement is reasonably accurate.

4. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should complete the oversight reviews for at least three high-risk local educational consortia or local governmental agencies by December 31, 2015, and post the results to its website.

5. Until the Legislature implements our recommendation in Chapter 2, Health Care Services should immediately resolve weaknesses in its oversight of local educational consortia and local governmental agencies to ensure that these entities sufficiently meet their responsibilities under the administrative activities program and meet the terms of their contracts with Health Care Services. Health Care Services should complete the oversight reviews for any remaining high-risk local educational consortia or local governmental agencies by June 30, 2016, and post the results to its website.

8. To minimize the risk that claiming units could include unallowable costs when calculating their reimbursement claims, Health Care Services should remind all local educational consortia and local governmental agencies that contracts with their claiming units should prohibit claiming units from seeking federal reimbursement of Health Care Services' participation fee.

10. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and develop and implement a plan to take over responsibility for conducting quarterly time surveys and performing related activities as soon as reasonably possible.

11. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and develop and issue a request for proposals to identify a responsible vendor to assist in implementing a statewide quarterly random moment time survey.

12. To streamline the organizational structure of its administrative activities program and to improve the program's cost-effectiveness, Heath Care Services should implement a single statewide quarterly random moment time survey and draft revisions to regulations as appropriate and to applicable documents, including the manual, oversight strategies and plans, and policy and procedure letters.

13. To the extent that local educational consortia and local governmental agencies are no longer involved in the administrative activities program, Health Care Services should develop and issue a standard contract for claiming units to sign to participate in the program.

17. To better maximize federal reimbursements for the administrative activities program, Health Care Services should, within six months, develop and implement a method to oversee and track the outreach efforts that local educational consortia and local governmental agencies use for ensuring that nonparticipating claiming units understand the benefits and consider participating in the administrative activities program.

18. To better maximize federal reimbursements for the administrative activities program, Health Care Services should, within six months, revise reimbursement rates to authorize claiming units to claim the 75 percent reimbursement rate for translation activities as federal law allows.

19. To better maximize federal reimbursements for the administrative activities program, Health Care Services should, within six months, determine the extent to which claiming units can claim the unreimbursed difference between the 50 percent and 75 percent reimbursement rates for translation activities for past years and inform claiming units of the findings.

20. Should the Legislature implement our recommendation in Chapter 2 to allow claiming units to submit reimbursement claims directly to it, Health Care Services should develop and implement its own outreach functions to ensure that claiming units that do not currently participate understand the benefits and consider participating in the administrative activities program.

21. To provide the public with the ability to participate fully in developing the rules governing the administrative activities program, Health Care Services should, in accordance with California's Administrative Procedure Act (APA), immediately develop and adopt the regulations cited in the four subdivisions of Section 14132.47 of the California Welfare and Institutions Code.

23. To ensure that it provides stakeholders with timely access to information regarding the billing option program, Health Care Services should issue the required annual report covering April 2013 to May 2015 by December 2015 as promised.

24. To ensure that it provides stakeholders with timely access to information regarding the billing option program, Health Care Services should issue all future annual reports in a timely manner.

6. To reduce costs for unnecessary evaluations, Mental Health should either issue a regulation or seek a statutory amendment to clarify that when resolving a difference of opinion between the two initial evaluators of an offender, Mental Health must seek the opinion of a fourth evaluator only when a third evaluator concludes that the offender meets SVP criteria.

3. To promote consistency and ensure that it provides sufficient guidance to evaluators, State Hospitals should update its assessment protocol by March 2016 to include more specific instructions on how to conduct evaluations, such as what assessment instruments evaluators may use and what documents they should consider. State Hospitals should also develop a timeline for periodically reviewing and making any necessary updates to the assessment protocol.

4. To comply with state law, State Hospitals should ensure that it follows the Administrative Procedures Act for future changes to its standardized assessment protocol.

14. State Hospitals should explore options for tracking the time evaluators spend on each evaluation activity to increase the accuracy of the workload equivalencies it includes in its workload matrix and should implement such options by September 2015.

11. To fulfill its charge to evaluate MHSA programs, the Accountability Commission should undertake the evaluations specified in its implementation plan.

3. To maximize the usefulness of the information it makes available to stakeholders and to increase accountability, the board should create policies and procedures that include clear, comprehensive guidance to counties about all aspects of performance outcome and expenditure reporting. At a minimum, such guidance should include specifying how counties should define when a juvenile has received a service and whether certain services, such as training, should qualify as serving juveniles.

5. To maximize the usefulness of the information it makes available to stakeholders and to increase accountability, the board should consider verifying the counties' data by conducting regular site visits on a rotating basis or by employing other procedures to verify data that counties submit.

12. To ensure that counties do not maintain excessive balances of unexpended block grant funds, the board should develop procedures to monitor counties' unspent funds and follow up with them if the balances become unreasonable.

5. To ensure that it can better monitor how its medical staff and contractors adhere to the informed consent requirements of Title 22, sections 70707.1 through 70707.7, the Receiver's Office should develop a plan by August 2014 to implement a process by December 2014 that would include working with Corrections to establish a process whereby inmates can have witnesses of their choice when consenting to sterilization, as required by Title 22, or working to revise such requirements so that there is an appropriate balance between the need for secure custody and the inmate's ability to have a witness of her choice.

11. To ensure that inmates receive only medical services that are authorized through its utilization management process, the Receiver's Office should ensure that the computer system it procures includes functionality to electronically link medical scheduling with authorization through the utilization management process to prevent all unauthorized procedures, regardless of whether they may result in sterilization, from being scheduled.

18. Corrections and Correctional Health Care should run a query of exempt positions related to the chief psychologist classification, such as clinical psychologists and senior psychologists, to determine whether any other exempt employees were improperly credited or paid for on-call or call-back assignments prior to December 2014, and seek recovery through reducing those employees' accumulated leave balances.

1. Corrections should take appropriate disciplinary actions against the employee and pursue collection efforts for the compensation she did not earn.

3. To eliminate duplicative effort and increase efficiency, Corrections should not make unnecessary referrals to Mental Health. For example, Corrections should better leverage the time and work it already conducts by including in its referral process (1) determining whether the offender committed a predatory offense, (2) reviewing results from any previous screenings and evaluations that Mental Health completed and considering whether the most recent parole violation or offense might alter the previous decision, and (3) using STATIC-99R to assess the risk that an offender will reoffend.

4. To eliminate duplicative effort and increase efficiency, Corrections should not make unnecessary referrals to Mental Health. Corrections and Mental Health should jointly revise the structured screening instrument so that the referral process adheres more closely to the law's intent.

1. To ensure that the State does not spend additional resources on COMPAS while its usefulness is uncertain, Corrections should suspend its use of the COMPAS core and reentry assessments until it has issued regulations and updated its operations manual to define how Corrections' use of COMPAS will affect decision making regarding inmates, such as clarifying how COMPAS results will be considered when sending inmates to different prison facilities, enrolling them in rehabilitative programs to address their criminal risk factors, and developing expectations for those on parole.

4. Adjust current employees' leave balances in the leave accounting system to correct any improper charging of leave identified by the audit.

6. In instances where the audit has determined that an employee's leave balance was mischarged but the employee subsequently departed state service, take appropriate measures to remedy any resulting incorrect compensation of the employee for unused leave upon his or her departure, including by seeking repayment of any amount overpaid to the employee.

11. To recoup the payment and leave accumulations to which its employees were not entitled, Corrections and Correctional Health Care should reduce the accumulated leave balances of Employee A by 886 hours. If his accumulated leave balances are not sufficient, offset any remaining hours against future accumulations of leave.

13. To recoup the payment and leave accumulations to which its employees were not entitled, Corrections and Correctional Health Care should work with the California Public Employees' Retirement System and attempt to recoup the $22,766 in CTO leave hours Employee B—who retired in April 2015—cashed out but to which he was not entitled.

19. Corrections and Correctional Health Care should run a query of exempt positions related to the chief psychologist classification, such as clinical psychologists and senior psychologists, to determine whether any other exempt employees were improperly credited or paid for on-call or call-back assignments prior to December 2014, and seek recovery through reducing those employees' accumulated leave balances.

1. To ensure that it collects fair market rents for the SR 710 properties on the State's behalf, Caltrans should, using the fair market rent determinations for all SR 710 properties it recently prepared and excluding those in its affordable rent program, adjust the tenants' rents to fair market after providing them with proper notice.

2. To ensure that it collects fair market rents for the SR 710 properties on the State's behalf, Caltrans should make only limited exceptions to charging fair market rent and document the specific public purpose that is served in any case that it does not charge fair market rent.

4. To remedy the effects of the improper governmental activity substantiated in this report and to prevent it from recurring, Caltrans should continue its efforts to recover the undercharged rent from the telecommunications companies.

3. To avert possible legal challenges, the Authority should ensure that the independent peer review panel adheres to the Bagley-Keene Open Meeting Act or seek a formal opinion from the Office of the Attorney General (attorney general) regarding whether the panel is subject to this act.

5. To ensure that the decal fee is sufficient to reimburse program costs, Motor Vehicles should periodically perform a full cost analysis of the decal program and update the fee accordingly

1. To ensure that programs supported by special plates receive appropriate amounts of revenues due to them, Motor Vehicles should annually collect all fees for special plates that are no longer on a vehicle but are retained by the plate owner.

3. Motor Vehicles should assess the extent to which it has charged fees for special plates that are not consistent with those prescribed in statutes and take appropriate action.

15. To make certain that money from the special plate funds pay only for allowable and supportable activities, Resources should use all appropriate funding sources to pay for any expenses that benefit multiple programs in proportion to the benefits these programs actually receive. Further, it should ensure that its allocation of such expenses to different funds is equitable and supported.

7. To ensure that the Legislature has the information necessary to meet the State's restoration goals and to plan for the State's future financial obligations related to mitigation, the Resources Agency should work with Fish and Wildlife and Water Resources to meet with the Legislature regularly to provide updates on the status of its restoration efforts and the feasibility study to ensure that the Legislature has the information necessary to make funding and other informed decisions.

8. To ensure that the Legislature has the information necessary to meet the State's restoration goals and to plan for the State's future financial obligations related to mitigation, the Resources Agency should work with Fish and Wildlife and Water Resources to develop an estimate of the costs, adjusted for inflation, that the State may incur for fulfilling its financial obligations related to mitigation under the QSA. The Resources Agency should include this information in the feasibility study so the Legislature is fully aware of the estimated costs and timing of the State's future financial obligations.

12. By February 2015 the Science Center should provide training to all of its personnel specialists on the number of leave hours employees earn for working on holidays.

1. When the commission determines that it will pursue delinquent lessees itself, it should use a collection agency or a program such as the Franchise Tax Board's Interagency Intercept Collections Program.

2. To ensure that it receives rent from the lessee that reflects the approximate value for the State's property at those times when a lessee disputes a modification to the rental amount after the commission exercises its right to perform a rent review or because the lease expired, the commission should include in its lease agreements a provision that requires lessees to pay the commission's proposed increased rental amount, which would be deposited into an account within the Special Deposit Fund. The increased rental amounts deposited, plus the corresponding interest accrued in the account, should then be liquidated in accordance with the amount agreed to in the final lease agreement.

21. To ensure that it manages delinquent leases in an effective and timely manner and collects all the amounts owed to it, the commission should develop and adhere to policies and procedures that incorporate the administrative manual’s guidance, including the steps staff should take when a lessee is delinquent, time standards for performing those steps, and a process for consistently tracking the status of delinquent leases between divisions.

4. To ensure that it can comply with state law in the event that it must close parks or reduce park services in the future, the department should improve its methodology for developing individual park unit budgets and determining and tracking park-level costs. Specifically, the department should develop specific time frames and deliverables for the completion of phases two and three of its plan. These time frames should include specific completion dates for each key component of the phases.

6. To ensure that it can comply with state law in the event that it must close parks or reduce park services in the future, the department should improve its methodology for developing individual park unit budgets and determining and tracking park-level costs. Specifically, the department should determine how it will define service levels and measure whether those levels are being met so it can provide budgets for each park unit, as phase three of its process requires.

8. To prevent unauthorized leave buyback transactions, the department should provide training by December 2013 to all department managers and personnel staff who might be involved in leave buyback transactions to ensure that they understand the State's requirements regarding leave buybacks.

11. To prevent unauthorized leave buyback transactions, the department should limit access for keying transactions to the payroll system only to authorized personnel staff.

15. To improve the effectiveness of the EPRC, the department should establish a process by March 2014 through which the director's office provides formal direction to the EPRC regarding staffing priorities.

4. To provide the public with accurate and complete information on the bond-funded projects it administers, Water Resources should develop and consistently use a formalized, documented review process that will provide greater assurance that project information posted to the Bond Accountability website is regularly updated and contains accurate information.

2. To ensure it can demonstrate that its fraud prevention efforts are maximizing financial recoveries for the beverage program, CalRecycle should both modify and annually update its fraud management plan to include the following:
- By December 31, 2014, formally establish a systematic process for analyzing, monitoring, and responding to the risk of fraudulent recycling of out-of-state beverage containers.
- Develop fraud estimates—by type of fraudulent activity—that quantify the potential financial losses to the beverage program and the methodology CalRecycle used to develop these estimates.
- Identify the amount of actual fraud in the prior year by type of fraudulent activity, such as the financial losses resulting from the redemption of out-of-state beverage containers or the falsification of reports used to substantiate program payments.
- Identify the amount actually recovered for the beverage program in the form of cash for restitution and penalties resulting from fraud.

3. To allow for public input and to prevent any legal challenges claiming that its policies and procedures regarding prepayment holds constitute unenforceable underground regulations, CalRecycle should adopt these policies and procedures as regulations in accordance with the Administrative Procedure Act.

5. To ensure it effectively uses resources, CalRecycle should renegotiate its agreement with the California Department of Justice (Justice) to pay based on the cases CalRecycle refers, limiting costs to investigations over a predefined period of time. CalRecycle could agree to increase the budget for a particular investigation if Justice demonstrates that it has developed promising leads.

6. When regional water boards include staff enforcement costs in the penalty actions they issue, the state water board should require that they use a systematic method for tracking the hours staff spend on enforcement activities related to penalty actions and maintain documentary support for these staff enforcement cost calculations.

7. If regional water boards continue to include staff enforcement costs in the penalty actions they issue, the state water board should revise its staff cost rate to reflect actual staff salaries and overhead cost for the certification program.

30. To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should formally track and monitor the timeliness of its processing of applications by type and the cause of any delays.

32. To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe in order to identify the workload capability of each of its units, such as the licensing support unit; to the extent it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified.

33. To ensure that BRN continues to process applications within regulatory time frames, it should continue its efforts to refine its business processes to increase efficiency and reduce the amount of time applications are pending its review.

7. To ensure that it adequately tracks critical information related to its basic functions and mission, the commission should work with Consumer Affairs to ensure that the new online program will meet its needs and requirements. Once the program is in place, the commission should use it as its central means for tracking its operations.

9. To ensure that it accurately collects revenue, the commission should calculate the pension assessment by counting all the complimentary tickets issued, except for working complimentary tickets, not merely the complimentary tickets that are redeemed. If the commission does not agree that it should calculate the pension assessment by counting all the complimentary tickets issued, it should seek a change in its regulations to calculate the fee based only on the number of complimentary tickets redeemed.

22. The commission needs to establish regulations that describe its process for determining its ticket assessment for the neurological account so that it avoids the use of underground regulations.

28. To comply with state law governing the pension plan, the commission needs to, after it has an accurate and complete listing of all licensed athletes and box office information by event type, conduct the analysis to determine the feasibility of expanding the pension plan to cover all athletes and report the results to the Legislature.

17. To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost-benefit analysis to ensure that BreEZe is a cost-effective solution to meet these regulatory entities' business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost-benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change.

35. To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented.

36. To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should work with the regulatory entities to develop training that is specific to each entity's business processes.

10. To comply with state law and to ensure that it effectively manages its inspections of institutions, the bureau should prioritize its announced and unannounced inspections to focus on those institutions that have a higher risk of noncompliance.

1. The physical therapy board should explore the feasibility of establishing a state position to perform the duties of its current in-house consultant at a reduced cost.

3. To ensure that the state board assesses the merits of pursing federal funding for workforce investment programs in California, it should establish procedures by December 31, 2015, that include, at a minimum, the following:
- The methods it will use to identify federal grant opportunities.
- The factors it will consider in its decisions to pursue or forgo applying for these grants.
- The process by which it will document its analyses of grants and its final decisions to either pursue or forgo grant opportunities.
- A method for ensuring that the process is consistently implemented.

4. To ensure that the State maximizes federal funding opportunities related to workforce investment, EDD and the state board should formalize their collaborative grant-seeking procedures by December 31, 2015, to clearly define their respective roles and responsibilities in the grant-seeking process.

4. As part of an overall strategy to limit the number of calls it receives while still providing timely and effective customer service, the department should use existing data and additional data from the new phone system to gain a better understanding of why people request to speak to an agent. Using this information, the department should further develop strategies and measurable goals related to achieving a reduction in call volumes. For example, to ensure that virtually all calls are able to gain access to the voice response portion of its new phone system, the department should monitor the volume of blocked call attempts and work with its phone system vendor if necessary to increase the system's capacity.

5. To evaluate the effectiveness of its other efforts to provide services to claimants in ways that do not require them to speak to agents, such as Web-Cert and Tele-Cert, the department should periodically summarize and assess the more robust management information available under its new phone system.

4. To assist the state board and other workforce investment partners in the development and implementation of state-specific performance measures, EDD should ensure that it works with the state board to develop procedures for approving the addition of data elements to its Web-based system and for the exchange of data between EDD and the state board.

5. To identify ways to better serve veterans in California, the department should assess the success or struggles of veterans within demographic categories in finding employment, such as age, race, or educational attainment, by comparing veterans' performance to that of nonveterans in the same demographic categories and across demographic categories and use this analysis to determine whether specific populations of veterans could be better served through more targeted efforts and to identify best practices for improving employment outcomes for these specific populations. Further, the department should provide the results of this analysis annually, beginning in 2014, to stakeholders, including local workforce agencies, the state workforce board, the interagency council, the Legislature, and the public.

8. To identify and correct any policies, procedures, or practices that may be contributing to avoidable appeals filed by claimants and employers and thereby provide eligible claimants with unemployment benefits in a timelier manner, EDD should do the following: Using the appeals board's data from fiscal year 2013-14, EDD should identify the legal isssues where its determinations are most frequently overturned, and use these data to establish initial performance benchmarks. In addition, similar to the review that EDD's audit and evaluation division performed in 2012, EDD should then review samples of its overturned determinations and the appeals board's decisions on these legal issues to identify trends in the reasons the appeals board cites for overturning EDD's determinations. With this information, EDD should review its policies, practices, and training related to these areas and identify and correct any weaknesses that may be contributing to the overturning of determinations. By April 1, 2015, EDD should report to the Legislature on the results of this review and any changes it plans to make to its determination process.

9. To identify and correct any policies, procedures, or practices that may be contributing to avoidable appeals filed by claimants and employers and thereby provide eligible claimants with unemployment benefits in a timelier manner, EDD should do the following: EDD should use the semiannual data that the appeals board provides to determine whether changes it makes to its process result in reductions in the percentage of its determinations that are overturned on appeal. EDD should also review these data to determine whether it needs to conduct additional reviews of its determinations and the appeals board's decisions to identify additional opportunities for improvement. EDD should report these results to the Legislature annually.

1. To ensure that its grant-seeking process is effective in considering grant opportunities related to workforce investment, EDD should update its grant identification and analysis procedures by December 31, 2015, to include the following:
- Identification of the appropriate level of management or staff to analyze and document the factors considered in pursuing or forgoing grant opportunities.
- Documentation of grant analyses, including factors considered in pursuing or forgoing grant opportunities.
- Documentation of decisions related to pursuing or forgoing grants.
- A method for ensuring that the process is consistently implemented.
- Training of appropriate management or staff about EDD's grant-seeking process.

2. To ensure that the State maximizes federal funding opportunities related to workforce investment, EDD and the state board should formalize their collaborative grant-seeking procedures by December 31, 2015, to clearly define their respective roles and responsibilities in the grant-seeking process.

2. To correct the erroneous leave hours we identified in our analysis of the leave accounting system, CalHR should work with the state controller and all state agencies under its authority to review and take the appropriate action to correct the errors by January 2015.

7. To ensure that state agencies accurately account for their employees' leave benefits, CalHR should consolidate guidance by January 2015 regarding the appropriate amount of leave that employees should earn each month and provide these criteria to the state controller to use when developing the leave accounting system's monthly exception reports. For example, CalHR should identify the number of holiday credit hours that employees covered by each collective bargaining agreement should receive for working on a holiday.

9. To ensure that state agencies accurately account for their employees' leave benefits, CalHR should establish general parameters and issue guidance to state agencies by January 2015 on how to account for the leave hours for employees who work alternate work week schedules.

10. To ensure that state agencies accurately account for their employees' leave benefits, CalHR should provide additional guidance to state agencies by January 2015 on interpreting the provisions of the collective bargaining agreements related to the amount of leave employees earn. For example, CalHR could provide scenarios to illustrate the number of hours employees should earn under common circumstances.

11. To ensure that IT projects have the oversight needed to better position them for success, Technology should require state departments to follow its IT policies, including developing all necessary plans and receiving all required training.

12. To ensure that IT projects have the oversight needed to better position them for success, Technology should ensure that departments obtain IPO and IV&V services as soon as an IT project is approved. Additionally, Technology should document key discussions with any department in which the IV&V consultant and the IPO specialist raise significant concerns about a project, and ensure that the respective department responds promptly to, and adequately addresses, the concerns that the IPO specialist and the IV&V consultant raise.

13. To ensure that IT projects have the oversight needed to better position them for success, Technology should require state departments to appropriately address deficiencies identified in Technology's approval of any SPRs as conditions that must be met, and to do so according to specified timelines. If the deficiencies are not adequately addressed within the specified timelines, Technology should take action to suspend the project until the department has either resolved the identified deficiencies or adequately documented its justification for not addressing the deficiencies.

14. To ensure that IT projects have the oversight needed to better position them for success, Technology should develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department's IT project reaches or exceeds these thresholds, Technology should require the department to conduct a cost-benefit analysis for the project and include this analysis in an SPR. Technology should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction.

28. To ensure that contracting departments comply with requirements, Technology should periodically monitor the status of contracts and enforce adherence to procurement requirements. For example, Technology should ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract.

1. By December 2015, Technology should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and how IPO analysts should recommend corrective action and escalate issues to Technology's management.

2. By December 2015 Technology, should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including when and what Technology should require that sponsoring agencies perform as remedial actions, and what sanctions Technology will impose for noncompliance with these remedial actions.

3. By December 2015 Technology, should develop and adopt criteria to guide the type and degree of intervention it will take to prevent IT projects with significant problems from continuing without correction, including what conditions could trigger Technology to consider suspending or terminating an IT project.

4. To clarify and reinforce its oversight authority with sponsoring agencies, by December 2015 Technology should develop a method to formally document and communicate its expectations with the sponsoring agencies whose projects are under Technology's oversight.

6. To help ensure the independence and objectivity of IPO analysts working in the oversight and consulting division, Technology should provide regular training regarding maintaining independence while conducting project oversight.

9. To ensure that the sponsoring agencies' project status reports provide a reliable and consistent assessment of an IT project's progress, Technology should develop and adopt specific standards that describe how to calculate and report the project's current status.

10. To attract and retain employees with appropriate experience and qualifications to perform IT project oversight, Technology should continue its efforts to gain approval to modify and use the project manager classification for the IPO analyst role.

13. To improve its oversight training, by June 2015 Technology should continue to implement a consistent and repeatable training plan for IPO analysts, which includes contract management, project assessment, IT systems engineering, and maintaining independence.

3. To assist reporting entities in reaching full compliance with the security standards, the technology department should ensure the consistency and accuracy of its self certification process by developing a self assessment tool by December 2015 that reporting entities can use to determine their level of compliance with the security standards. The technology department should require reporting entities to submit completed self assessments along with their self certifications.

4. To assist reporting entities in reaching full compliance with the security standards, the technology department should provide more extensive guidance and training to reporting entities regarding the self certification process, including training on how they should use the new self assessment tool.

8. The technology department should revise its certification form to require reporting entities to submit detailed information about their compliance with the security standards. It should use this information to track and identify trends in the State's overall information security.

9. The technology department should develop policies and procedures to define the process and criteria it will use to incentivize entities' compliance with the security standards.

10. To improve the clarity of the security standards, the technology department should take the following actions: Perform regular outreach to all reporting entities to gain their perspectives, identify any unclear or inconsistent security standards, and revise them as appropriate.

11. To improve the clarity of the security standards, the technology department should develop and regularly provide detailed training on the requirements of the security standards and on best practices for achieving compliance. It should provide these trainings in a variety of locations and formats, including webinars.

1. To ensure public safety and provide public assurance that school districts construct projects in accordance with approved plans, the department, in conjunction with the division, should pursue legislative changes to the Field Act that would prohibit occupancy in cases in which the division has identified significant safety concerns.

37. To ensure that the construction unit complies with the State's procurement laws and policies, General Services should continue its efforts to implement regulations that govern the small business certification process related to defining and enforcing violations of commercially useful function requirements.

9. No later than December 1, 2015, Franchise Tax Board should develop a plan to determine whether the accessibility violations we identified exist on other portions of its online presence that we did not include in the scope of our review. Once this plan is executed, it should correct violations wherever it finds them and do so no later than June 1, 2016.

5. To better utilize unused space at the veterans homes, and to serve more veterans within legal restrictions, CalVet should use the information in the Yountville veterans home master plan to develop a plan for using unused space at that home to generate revenue and/or serve more veterans.

11. To ensure it is maximizing its ability to serve veterans in the State's veterans homes, CalVet should follow through with its plan to assess the bed capacity of the homes for veterans' needs after the homes in Fresno and Redding are licensed, to determine the most appropriate number of beds for the different levels of care offered at each home.

15. To allow for public input and to prevent any legal challenges that its policy of capping members' fees, its $165 monthly income-retention policy for members, and its age and admission policy are unenforceable underground regulations, CalVet should adopt these policies as regulations in accordance with the Administrative Procedure Act.

17. To more effectively and efficiently meet state purchasing and procurement requirements, CalVet should continue implementing quality assurance policies to strengthen its oversight of its purchasing practices, including conducting on-site reviews of the purchasing practices of the veterans homes.

1. CalVet should fully implement the recommendations from our 2009 report.

2. Veterans Services should develop and implement a plan by December 31, 2015, to routinely analyze and use myCalVet data to identify trends in the services veterans and their families indicate they are most interested in and incorporate the results of such analyses into its outreach efforts.

3. Establish a routine of completing a monthly inventory of expendable state property after the barcode system is implemented.

2. The commission should determine the cause of its lack of compliance with state law requiring it to issue award decisions within 75 days of the date an intervenor submits a compensation claim, and it should determine what actions to take to rectify the problem. The commission should ensure that it has sufficient information, such as detailed tracking information regarding claims, to identify where in the process delays are occurring. If the commission determines that the current 75-day statutory period is unreasonable, it should seek a change in state law.

5. To comply fully with state law, the commission should conduct a comprehensive market rate study and update it periodically.

14. The commission should follow the requirement in state law to inspect and audit the accounting records of utilities it regulates within required time frames. If the commission chooses to continue to meet this requirement through the general rate case process, it should ensure that all utilities file a general rate case on a regular schedule so as to comply with the state law's audit requirement. However, the commission should follow alternate methods to comply with the audit requirement when a utility will not be filing for its general rate case in time to be audited within three or five years, depending on the timing of the required audit for that utility.

2. To ensure that it resolves complaints against passenger carriers in a timely manner, the commission should establish a method for prioritizing complaints and it should implement a policy specifying the maximum amount of time between the receipt of a complaint and the completion of the subsequent investigation. Further, the commission should require branch management to monitor and report regularly on its performance in meeting that policy.

6. To ensure that the branch conducts thorough investigations of passenger carriers, the commission should implement a formal training program to ensure that all investigators have adequate knowledge and skills related to regulating passenger carriers.

9. To ensure that passenger carriers submit accurate fee payments, the commission should require its fiscal staff to implement a process to verify passenger carrier fee payments and associated revenue.

10. To ensure that it complies with state law and uses passenger carrier fees appropriately, the commission should implement a process to ensure that passenger carrier fee revenues more closely match related enforcement costs.

12. To detect and deter carriers from operating illegally at airports, the branch should use as intended the five positions added for passenger carrier enforcement at airports. If the branch chooses not to designate five positions solely for this purpose, then it must be prepared to demonstrate regularly that an equivalent number of full-time positions are working on this activity.

13. To strengthen its leadership and ensure passenger carrier and public safety, the branch should produce a draft strategic plan by December 31, 2014, with a final strategic plan completed as the commission specifies. The strategic plan should include goals for the program; strategies for achieving those goals, including strategies for staff development and training; and performance measures to assess goal achievement.

3. To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should continue to implement its quality management team program component focused on reviewing the categorization of complaints and correcting identified errors.

4. To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should develop and implement tools by September 30, 2015, to measure the quality management team program's effectiveness.

11. The commission should ensure that it complies with all policy requirements in SAM Chapter 5300 no later than April 2016.

12. As part of developing, implementing, and maintaining an entitywide information security program, the commission should complete and maintain inventory of all its information assets, specifically categorizing the level of required security of the information assets based on the potential impact that a loss of confidentiality, integrity, or availability of such information would have on its operations and assets.

13. As part of developing, implementing, and maintaining an entitywide information security program, the commission should develop a risk management and privacy plan and conduct an assessment of risks facing its information assets.

14. As part of developing, implementing, and maintaining an entitywide information security program, the commission should develop, implement, and maintain an information security plan as part of its entitywide information security program.

15. The commission should develop, disseminate, and maintain an incident response plan.

16. The commission should revise its existing recovery plan to include a list of applications supporting critical business functions, their maximum acceptable outage time frames, and detailed recovery strategies for each application.

17. The commission should revise its existing recovery plan to include detailed procedures for rebuilding its technology infrastructure at an alternate processing site.

18. The commission should conduct regular tests and exercises to assess the sufficiency of the revised recovery plan and refine the plan when necessary.

1. To make certain that the research program contributes to the goals of the solar initiative, the commission should conduct a program evaluation before the remaining grant projects are completed.

3. To show how air pollution emissions reductions related to the solar initiative benefit the State, the commission should include in future reports the measurable benefits of those reductions.

1. To enhance the value of the Bond Accountability website, Finance should require administering agencies to provide information about the actual amounts of bond funds spent on posted projects at least semiannually.

2. To enhance the value of the Bond Accountability website, Finance should develop a tracking and review process to periodically assess the completeness of the project information posted to the website. Such a process should include a review of whether state agencies are describing, in terms the public can easily understand, the expected or realized benefits of bond-funded projects.

7. To further its mission to obtain the lowest possible rates for reliable and safe utility service for ratepayers through its reviews of balancing accounts, Ratepayer Advocates should use the commission's list of balancing accounts to guide its selection of the number, size, and type of balancing accounts to review so that its review coverage is more proportional across all utilities.

2. To improve the usefulness of the Judicial Council's semiannual reports until a statutory requirement is enacted, the AOC should work with the Judicial Council to pursue a cost-effective method to provide the semiannual reports in an electronic format that can be read by common database and spreadsheet software products that allow users to readily sort and filter the data, beginning with the semiannual report covering the July 1, 2013 through December 31, 2013, reporting period.

3. To improve the usefulness of the Judicial Council's semiannual reports until a statutory requirement is enacted, the AOC should work with the Judicial Council to pursue a cost-effective method to do the following:

- Include new contracts and the complete history of contracts amended during the reporting period in the semiannual reports, including the date of the original contract; the original contract amount and duration; all subsequent contract amendments; and the date, amount, and duration of each such amendment. The AOC should present this information beginning with the semiannual report covering the July 1, 2014, through December 31, 2014, reporting period.

4. To improve the usefulness of the Judicial Council's semiannual reports until a statutory requirement is enacted, the AOC should work with the Judicial Council to pursue a cost-effective method to do the following:

- Begin tracking additional information in its data systems for inclusion in the semiannual reports. This information should include whether a contract was competitively bid, the justification if it was not competitively bid, and whether the contract was with a Disabled Veteran Business Enterprise. For information technology contracts, the AOC should identify whether the contract was with a small business. The AOC should present this information beginning with the semiannual report covering the July 1, 2014, through December 31, 2014, reporting period.

5. The AOC should implement all of the best practices related to general and business process application controls as outlined in the U.S. Government Accountability Office's Federal Information System Controls Audit Manual no later than December 31, 2014, thereby strengthening and continuously monitoring the effectiveness of the controls over its information systems. In addition, the AOC should immediately begin implementing improvements to its controls over access to its information systems and place these improvements into effect by February 2014. Finally, the AOC should provide guidance and routinely follow up with the superior courts—requiring updates every six months until all identified issues are corrected—to ensure that they make the necessary improvements to their general and business process application controls.

6. The AOC, the Supreme Court, and the first, second, and fourth districts should implement procedures to ensure that they follow a competitive process for their procurements when required.

13. The AOC should revise the judicial contracting manual to require judicial entities to maintain documentation on their determinations of fair and reasonable pricing for purchases under $5,000.

17. The AOC should revise the judicial contracting manual to require that judicial entities maintain documentation for their evaluation and selection process used for competitive procurements. The AOC should also strengthen its procedures to ensure that bid evaluations are conducted properly and calculated correctly.

20. The AOC, HCRC, Supreme Court, and fourth and fifth districts should implement procedures to ensure that required noncompetitive procurement processes, such as preparing justifications and obtaining approval for sole-source procurements, are properly documented. Additionally, the AOC should ensure that it prepares the appropriate documentation when it amends a contract that it has competitively solicited and the amendment includes a change that was not evaluated in the original competitive process.

25. The AOC should implement procedures to ensure that its internal controls over payments are followed and that procurements are approved before ordering and receiving goods and services.

3. To ensure that its compensation structure is reasonable, the AOC should mirror the executive branch's practices for offering leave buyback programs in terms of frequency and amount.

4. To increase its efficiency and decrease its travel expenses, the AOC should require its directors and managers to work in the same locations as the majority of their staff unless business needs clearly require the staff to work in different locations than their managers.

11. To reduce its expenses, the AOC should cease its excessive reimbursements for meals by adopting the executive branch's meal and travel reimbursement policies.

14. To ensure that it spends funds appropriately, the AOC should develop and implement controls to govern how its staff can spend judicial branch funds. These controls should include specific definitions of local assistance and support expenditures, written fiscal policies and procedures as the rules of court require, and a review process.

19. The AOC should conduct a comprehensive survey of the courts on a regular schedule—at least every five years—to ensure that the services it provides align with their responses. The AOC should re-evaluate any services that the courts identify as being of limited value or need.

20. To justify its budget and staffing levels, the AOC should conduct the steps in CalHR's workforce planning model in the appropriate order. It should begin by establishing its mission and creating a strategic plan based on the needs of the courts. It should then determine the services it should provide to achieve the goals of that plan. The AOC should base its future staffing changes on the foundation CalHR's workforce planning model provides. Finally, the AOC should develop and use performance measures to evaluate the effectiveness of this effort.

23. To reduce the risk that it may not identify an armed prohibited person, Justice should revise its electronic matching process to use all personal identifying numbers available in its databases.

24. To ensure that timely information is available for its efforts to identify armed prohibited persons and confiscate their firearms, Justice should manage staff priorities to meet both its statutory deadline for firearms background checks and its internal deadline for initially reviewing potential prohibited persons. Justice should report annually to the Legislature about the backlog of unreviewed potential prohibited persons and what factors have prohibited it from efficiently reviewing these persons.

34. To ensure that it fully supports its decision to apply federal prohibition terms to individuals, Justice should review all applicable federal and state laws and continue to seek clarification from the ATF and any other appropriate federal agencies to determine whether California's firearms restoration process meets federal criteria and, if not, why it does not. Justice should issue a report to the Legislature, within one year, detailing the results of its review and, if applicable, communicate why California's restoration process does not meet federal criteria and the impact that it has on prohibited persons who live in California.

4. To report to the Legislature about the effectiveness of its RADS program and to better inform decisions about expanding the number of analyzed sexual assault evidence kits, Justice should amend its agreements with the counties participating in the RADS program to require those counties to report case outcome information, such as arrests and convictions for the sexual assault evidence kits Justice has analyzed under the program. Justice should then report annually to the Legislature about those case outcomes.

1. To make certain that emergency response agencies receive reimbursements on time, Cal EMA should establish procedures to ensure that paying entities do not delay reimbursements.

7. If FEMA determines that the calculations and claims identified in the Office of Inspector General's audit report were erroneous, Cal EMA should modify the time sheets to track the actual hours that the responding agency works as well as the dates and times that the agency committed to the incident and returned from the incident.

8. If FEMA determines that the calculations and claims identified in the Office of Inspector General's audit report were erroneous, Cal EMA should ensure that the replacement for its current invoicing system can calculate the maximum number of reimbursable personnel hours under both FEMA's policy and the CFAA.

11. To make certain that money from the special plate funds pay only for allowable and supportable activities, Cal EMA should maintain documentation to support its charges to the antiterrorism fund. For example, it should ensure that employees submit signed time reports to support the time they spend on antiterrorism-related activities.

1. The authority should amend its regulations to bring them into accord with the 2004 act, thus allowing any eligible hospital to apply for the 2004 act's funds that remained as of June 30, 2014.

1. To ensure that the compensation the AOC provides is reasonable, the Judicial Council should adopt procedures that require a regular and thorough review of the AOC's compensation practices including an analysis of the job duties of each position to ensure that the compensation aligns with the requirements of the position. This review should include comparable executive branch salaries, along with a justification when an AOC position is compensated at a higher level than a comparable executive branch position.

15. The Judicial Council should develop rules of court that create a separate advisory body, or amend the current advisory committee's responsibilities and composition, that reports directly to the Judicial Council to review the AOC's state operations and local assistance expenditures in detail to ensure they are justified and prudent. This advisory body should be composed of subject matter experts with experience in public and judicial branch finance.

21. To ensure that it provides services to the trial courts as efficiently as possible, the Judicial Council should explore implementing a fee-for-service model for selected services. These services could include those that are little used or of lesser value to the trial courts, as identified in our survey that we discuss in Chapter 3.

22. To justify the budget and staff level of the AOC, the Judicial Council should implement some or all of the best practices we identified to improve the transparency of AOC spending activities.

4. To enhance the value of the HAVA spending plan as a transparency and accountability tool for the Legislature, the Office should make the following modifications to its annual HAVA spending plan:

Clearly state the methodology used to report prior HAVA expenditures in the HAVA spending plan. Such a methodology should use the financial information contained in its accounting system.

Reconcile the prior HAVA expenditures with the year-end financial reports the Office provides to the California State Controller's Office.

Present prior HAVA expenditures by activity and by specific appropriation.

5. To ensure the State complies with the NVRA, the Office should take all necessary steps, including seeking any necessary legislative changes, and work with the DMV to modify the driver's license application so that it may simultaneously serve as a form for voter registration.

7. To ensure that the audit and review unit's random audits of closed case files provide an effective oversight mechanism, the State Bar should follow its policy to conduct and record meetings and trainings related to the audit report's recommendations.

4. To ensure that it can accurately estimate any shifts in worker productivity and state revenue, BOE should strengthen its current methodology by analyzing the productivity and revenue collection of its employees and by monitoring those metrics at least semiannually. Additionally, BOE should support its methodology with documentation.

4. To improve the accuracy of information in the leave accounting system and to ensure that agencies do not improperly credit employees with leave in the future, the state controller should implement additional controls by June 2015 to prevent the leave accounting system from processing the types of inappropriate transactions we identified in our statewide electronic analysis. For example, it could develop cost-effective controls in the leave accounting system that would prevent employees from receiving annual leave and sick leave during the same pay period.

6. To make certain that the FCS evaluators are qualified, the Sacramento family court should develop processes to ensure that it signs all FCS evaluator declarations of qualifications annually.

7. To make certain that the FCS evaluators are qualified, the Sacramento family court should ensure that its unlicensed FCS evaluators complete the licensing portion of the annual declarations of qualifications.

8. To make certain that the FCS evaluators are qualified, the Sacramento family court should identify the training each of the FCS evaluators need to satisfy the court rules' requirements and ensure that they attend the trainings.

10. To make certain that the FCS evaluators are qualified, the Sacramento family court should develop processes to ensure that evaluator declarations of qualifications include all relevant information, such as the evaluator's experience.

12. To make certain that the FCS evaluators are qualified, the Sacramento family court should take all reasonable steps to ensure its FCS evaluators meet the minimum qualifications and training requirements before assigning them to any future Family Code Section 3111 evaluations. If necessary, and as soon as reasonably possible, the court should require the FCS evaluators to take additional education or training courses to compensate for the minimum qualifications and training requirements that were not met.

16. To verify that its private mediator and evaluator panel members meet the minimum qualifications and training requirements before appointment, the Sacramento family court should obtain any missing applications and training records for private mediators and evaluators on its current panel list before appointing them to future cases.

20. To verify that its private mediator and evaluator panel members meet the minimum qualifications and training requirements before appointment, the Sacramento family court should reinstate its local rules for private mediators and evaluators to provide a minimum of three references, and for private evaluators to provide a statement that they have read the court's evaluator guidelines.

23. To make sure that the minor's counsel it appoints meet the additional standards required by the superior court's local rules, the Sacramento family court should obtain any missing applications for minor's counsel before appointing them to any future cases.

42. To strengthen its accounting process for California Family Code Section 3111 evaluations, the Sacramento Superior Court should update its accounting procedures related to billing FCS evaluation costs to include steps for verifying the mathematical accuracy of the FCS summary and the proper allocation of costs between the parties.