Report 2010-122 All Recommendation Responses

Report 2010-122: California Department of Transportation: Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine Its Performance Measures, and Improve Internal Controls (Release Date: April 2011)

Recommendation #1 To: Transportation, Department of

To improve accountability internally and with the public, Caltrans should create and incorporate an analysis of support cost budget variances in its quarterly report to the agency and in its annual report to the Legislature and the governor. The analysis should report on the number of completed projects with budget variances and on the number of open projects for which the estimates at completion predict budget variances. Further, the analysis should report on the overrun and underrun ratios for those projects, and the portions of the variances due to rates and hours. Also, Caltrans should include in its strategic plan a measurable goal for reducing variances.

Annual Follow-Up Agency Response From October 2013

The annual report was transmitted to the Legislature on November 15, 2012, and included the performance measure that targets support expenditures that are within a specified range of the support budget. The annual report can be found at http://www.dot.ca.gov/reports-legislature.htm.

As previously reported, Caltrans did not included in its report an analysis of variances due to rates or hours because the largest impact to the project costs are generally due to changes in scope of work such as additional permit requirements and mitigation requirements. As the audit report noted, the primary cause for cost overruns in the projects reviewed was the increase in hourly rates. Caltrans does not anticipate this being the case any time in the near future. However, Caltrans does perform a separate analysis of rates each year for planning purposes.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

As stated above, Caltrans included an analysis of overruns and underruns in its 2012 annual report to the Legislature. Further, we understand Caltrans' assessment of the ongoing usefulness of the rates-and-hours analysis. We acknowledged in the report on page 37 that a unique event--a substantial increase in engineer salaries following a union agreement--prompted our analysis to determine whether cost overruns were due primarily to the number of hours expended on projects or to the rate paid for those hours. We consider this recommendation fully implemented, but suggest Caltrans use this simple analysis periodically when there are significant changes to rates or hours.

Annual Follow-Up Agency Response From October 2012

This recommendation will be fully implemented when the report is transmitted to the Legislature on November 15, 2012. The performance measure that targets support expenditures that are within a specified range of the support budget was not included in the 2011 annual report to the Legislature due to timing issues, but will be included in the 2012 annual report.

Caltrans has not included in its report an analysis of variances due to rates or hours because the largest impact to project costs are currently due to changes in scope of work such as additional permit requirements and mitigation requirements. As the audit report noted, the primary cause for cost overruns in the projects reviewed was the increase in hourly rates. Caltrans does not anticipate this being the case any time in the near future. However, Caltrans does perform a separate analysis of rates each year for planning purposes.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #2 To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should devise, use, and publicize a consistent method for reporting the support-to-capital ratio on its Web site and in other reports to the public. Further, Caltrans should recalculate past support-to-capital ratios using the method devised to allow for comparison across years.

Annual Follow-Up Agency Response From October 2012

This recommendation was fully implemented in December 2011. However, Caltrans' status updates provided a combined response to 1.2.a and 1.2.b. The recommendation to 1.2.b was not fully implemented by the one-year status update. We provide a separate status for 1.2.b.

In response to recommendation 1.2.a, Caltrans updated the website with re-calculated ratios and a consistent methodology in December 2011. The website was updated in September 2012, with the 2012 results. Please see Attachment 2A-1 which consists of the Annual Overall Program Measure. The updated information can be found at the following link:

http://onramp.dot.ca.gov/hq/projmgmt/index.jsp?pg=106

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #3 To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should develop goals—and publicly report on the progress against those goals—for the support-to-capital ratio, based on project type—State Transportation Improvement Program or the State Highway Operation and Protection Program—and project size.

Annual Follow-Up Agency Response From October 2012

Caltrans established support-to-capital ratio goals based on project size. These goals and results can be found at the following link:

http://onramp.dot.ca.gov/hq/projmgmt/index.jsp?pg=106

Caltrans also developed an annual measure which includes all programmed STIP and SHOPP projects that completed construction during each fiscal year for which Caltrans was responsible for development and construction management. A copy of the Five Capital Cost Ranges is attached as Attachment 2B-1. This measure was published in the 3rd Quarterly Project Delivery Report. A copy of the report is included as Attachment 2B-2 and can also be found at the following link:

http://www.dot.ca.gov/hq/transprog/ctcbooks/2012/0612/033_3.9_Q3_Project_Delivery_Rept.pdf

The 4th Quarterly Project Delivery Report is included as Attachment 2B-3 and can be found at the following link:

http://www.dot.ca.gov/hq/transprog/ctcbooks/2012/0912/26_3.11.pdf

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Recommendation #4 To: Transportation, Department of

To improve performance metrics related to the support program, Caltrans should continue to explore the use of additional metrics, such as a measure based on a productivity index as described in a March 2011 draft study by the University of California, Davis.

Annual Follow-Up Agency Response From October 2013

Efforts to date on this recommendation included the results of the UC Davis Study and the Production Study modeled after the UC Davis Study. There were no useful applications for measurable metrics from either study. A copy of the Production Study was transmitted with the 2012 annual status.

After an additional year of evaluation, we concluded that these efforts do not have sufficient information to evaluate productivity due to the wide diversity of project factors (Locations, Environment, Type of Project, Design Strategy, Right of Way, etc.) that contribute to production hours needed to complete projects. At this time, Caltrans plans to continue its use of Support-to-Capital as a program-level performance metric based on project size.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Our recommendation was to continue to explore alternative approaches to measuring its performance. It appears Caltrans has done so and concluded that other measures it was reviewing were not adequate and that the support-to-capital ratio is still useful.

Annual Follow-Up Agency Response From October 2012

As Caltrans reported in the one-year status update, the additional metric was produced by the target date of July 2012. However, the Production Study identified four potential measures (a copy of the Production Study is attached.) Caltrans has reviewed the four measures and believes that measure four is a viable measure. Measure four is similar to the support-to-capital goal and uses hours to mitigate labor cost charges and indexes capital to mitigate capital escalation changes.

However, Caltrans is not prepared to adopt this as a full measure until the following is done:

• Additional years of data are needed to ensure the method reflects long-term production information.

• Additional analysis is needed to establish trends based on project groups that are based on the capital value.

• Additional analysis is needed to establish trends based project groups that are based on the capital value.

• Additional effort is needed to evaluate the pre-construction phase. The construction phase hours were evaluated first to see if there was any potential for a reasonable measure.

Once the issues discussed above are complete, Caltrans will have a better time estimate for full implementation of this recommendation. In the meantime, Caltrans is moving forward with the following timeline:

• Evaluate and determine a potential measure for project groups by capital value by January 2013.

• Evaluate and determine potential pre-construction measure by July 2013.

Caltrans believes that the effectiveness of a long-term measure will not be known for several years. The existing data sample was based on the past four years but additional years are necessary in order to have a complete set of data. Caltrans is also considering the impacts on productivity by the ongoing furloughs and the constraints of limited overtime hours.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #5 To: Transportation, Department of

To ensure that it monitors the status of projects, Caltrans should implement earned value management throughout its districts in a manner similar to the implementation in the Los Angeles district. To allow for performance evaluation of project work, Caltrans should ensure that these performance metrics are available at the task level for both active and completed projects. Caltrans should instruct districts to aggregate this information for all projects by task level, to better assess the effectiveness and efficiency of support costs by task level. Caltrans should also make available to project managers graphical displays of project cost and schedule performance.

Annual Follow-Up Agency Response From October 2013

Caltrans determined that full implementation of earned value management was contingent on the roll-out of the PRSM application. PRSM will provide earned value metrics and report at the task level for all projects in the system. The date for a complete roll-out of PRSM moved from June 2013 to September 2013 due to technical and training issues during the early district roll-out. Roll-out of PRSM was completed on August 29, 2013.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Caltrans' PRSM system is capable of generating information on earned value metrics such as a cost performance index and a schedule performance index. Further, Caltrans issued a directive to project delivery employees indicating that earned value management is a tool to manage projects' costs and schedules. The directive defines various earned value management terms and outlines who has responsibility for various elements of implementation.

Annual Follow-Up Agency Response From October 2012

As noted in Caltrans' one-year response, a Project Delivery Directive (PD-08) on Earned Value Management (EVM) was issued on April 5, 2012. PD-08 provided staff with a framework in terms of policy, definitions, roles and responsibilities for consistent implementation of EVM. Full implementation of EVM is contingent on the roll-out of the Project Resource and Schedule Management (PRSM) application which will provide earned value metrics and reports at the task level for all projects in the system. Caltrans is on track for a complete roll-out of PRSM by June 2013. Currently, Caltrans utilizes reports which graphically depict cost and schedule performance for all projects.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #6 To: Transportation, Department of

To better address costs associated with the support program, Caltrans should ensure that the PRSM system contains strong controls that ensure employees only charge time to projects and phases for which they are assigned.

Annual Follow-Up Agency Response From October 2013

Full implementation of PRSM was moved from June 2013 to September 2013 due to technical and training issues during the early district roll-out. Roll-out of PRSM was completed on August 29, 2013, and put in strong controls to ensure employees only charge time to projects their units are assigned.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Caltrans provided a training video explaining how PRSM works with Caltrans' time tracking system. The video includes a screen shot showing an error message a Caltrans employee would receive were the employee to attempt to charge time to a task to which the employee's unit was not assigned. Additionally, according to the video employees will not be able to charge time to a task unless that task is open by the manager in charge of the task. Caltrans appears to have controls in place to ensure only individuals assigned to projects may charge time to those projects; however, we have not conducted additional audit work to assess whether those controls are effective.

Annual Follow-Up Agency Response From October 2012

This recommendation will be fully implemented by June 2013, when the new Project Schedule and Resource Management (PRSM) system is fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #7 To: Transportation, Department of

To better address costs associated with the support program, Caltrans should commission an independent study of the costs and benefits of using consultants to address temporary increases in workload and, if the study reveals cost savings, use consultants. To the extent possible, Caltrans should also use temporary staff appointments for temporary increases in workload when consultants are unavailable.

Annual Follow-Up Agency Response From October 2013

As mentioned previously, two reports were prepared by CTC and Associates LLC (dated July 15, 2011, and October 28, 2011, respectively), which compared the use of in-house staff and consultants. In general, these reports concluded that cost should not be the overriding factor in deciding whether to outsource. Other factors such as expediting project delivery and managing workload should be taken into consideration when determining when and what work to outsource. Caltrans subsequently contracted an additional study to identify options or tools to improve decision-making processes regarding resource mix during workload peaks and valleys. Available resources (State staff, consultants, temporary help, overtime, students, retired annuitants, etc.) were identified along with recommendations on the best utilization of each of these resources. This final report was completed in May 2013, and was emailed to John Lewis.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Since our audit was released, Caltrans has commissioned several studies on the use of contractors. The first report concluded that recent studies on contracting support both cases—some that in-house staff were more cost-effective and others that consulting were more cost-effective. That report concluded that costs are not the overriding factor in the decision to outsource. The second report followed up on specific items identified in the first and did not modify the conclusions. A third report examined the use of contracting to balance workload and resources in several states. It noted that the states surveyed use contractors in a variety of capacities and that California uses in-house staff more than other states. The report also reiterated the point we discussed in our audit that California's Legislature has historically established a 10 percent to 90 percent, consultant-to-state staff ratio in Caltrans' budget. Finally, the report noted that the four states it surveyed, in addition to California, made limited use of temporary staff such as limited term staff, retirees, or students.

Annual Follow-Up Agency Response From October 2012

As reported in the one-year response, Caltrans Division of Research and Innovation contracted for an independent study to identify options or tools to improve decision-making processes regarding resource mix during workload peaks and valleys. This final report is expected to be completed by March 2013, at which time this recommendation will be fully implemented.

California State Auditor's Assessment of Annual Follow-Up Status: Not Fully Implemented

Recommendation #8 To: Transportation, Department of

To improve accountability internally and with the public, Caltrans should establish budgets for those STIP projects programmed before the passage of Senate Bill 45 so that overruns may be reported in the quarterly report to the agency and in the annual report to the Legislature and the governor.

6-Month Agency Response

Caltrans has established support budgets for the 24 projects it identified as having started (projects programmed) prior to the passage of Senate Bill 45. (2012-406, p. 194)

California State Auditor's Assessment of 6-Month Status: Fully Implemented

Recommendation #9 To: Transportation, Department of

To improve accountability internally and with the public, Caltrans should develop a system to report on the total budgets of support program projects—including initial project support budgets—of projects that have been divided into multiple projects or combined into a larger project.

6-Month Agency Response

Caltrans stated that it has developed improved business practices to allow for easier tracking of project budgets. Specifically, Caltrans provided a project management directive outlining a process for managing project funding and costs when projects are split or combined into one or more construction contracts. The process allows for tracking the origin of projects split into multiple projects or combined into one project. That directive took effect in August 2011. (2012-406, p. 194)

California State Auditor's Assessment of 6-Month Status: Fully Implemented

Recommendation #10 To: Transportation, Department of

To better develop and manage project budgets for support, Caltrans should instruct project managers to submit requests to update the budget when assumptions on which the budget was based are no longer valid, regardless of the phase of the project.

6-Month Agency Response

Caltrans issued a project management directive titled “Management of Capital Outlay Support,” in August 2011. The directive gives direction on updating budgets for construction on or before the date the project is voted on by the CTC and proceeds to the construction phase. Further, the directive includes instruction to update estimated hours in the project’s work plan when hours change and to review and update—if needed—resource estimates on an ongoing basis, and at least quarterly. Further, the directive requires that the project development team review and update support budgets at the completion of each major milestone. (2012-406, p. 195)

California State Auditor's Assessment of 6-Month Status: Fully Implemented

Recommendation #11 To: Transportation, Department of

To better develop and manage project budgets for support, Caltrans should direct its project managers to use a detailed approach based on project tasks, such as those included in a project work plan, when finalizing project support budgets before construction.

6-Month Agency Response

Caltrans issued a project management directive titled “Management of Capital Outlay Support,” in August 2011. The directive gives direction on updating budgets for construction on or before the date the project is voted on by the CTC and proceeds to the construction phase. Further, the directive includes instruction to update estimated hours in the project’s work plan when hours change and to review and update—if needed—resource estimates on an ongoing basis, and at least quarterly. Further, the directive requires that the project development team review and update support budgets at the completion of each major milestone. (2012-406, p. 195)

California State Auditor's Assessment of 6-Month Status: Fully Implemented

Recommendation #12 To: Transportation, Department of

To ensure that it monitors the status of projects, Caltrans should continue to implement the policies described in its February 2010 memorandum to the districts describing an approach Caltrans will take to monitor support costs within budget. Moreover, Caltrans should direct its project managers to monitor budgets for all projects according to both hours and costs.

6-Month Agency Response

Caltrans issued a project management directive in August 2011 clarifying the responsibility of project managers in the development and maintenance of project workplans, including planned hours and support costs throughout the life of the project. Further, Caltrans stated that it has added a standing agenda item to a quarterly teleconference to discuss support budget corrective action plans. (2012-406, p. 195)

California State Auditor's Assessment of 6-Month Status: Fully Implemented

All Recommendations in 2010-122

Agency responses received after June 2013 are posted verbatim.