Report 2015-121 All Recommendation Responses

Report 2015-121: California Department of Veterans Affairs: The State Paid Nearly $28 Million for a Flawed System That Fails to Meet the Needs of Its Veterans Homes (Release Date: June 2016)

Recommendation #1 To: Veterans Affairs, Department of

To ensure that its project management of IT projects promptly identifies potential problems and develops resolutions, by September 2016 CalVet should define the project executive's and project manager's responsibilities to ensure that the individuals who fill those positions take an active role in each project.

60-Day Agency Response

CalVet has developed CalVet IT Policy #ISD-16-001 which defines the project executive, project manager, and project team responsibilities. The new policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #2 To: Veterans Affairs, Department of

To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016 CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management and other required plans. This formal process should also include a process to periodically verify that the project team is adhering to all these plans.

60-Day Agency Response

CalVet has established a formal process through CalVet IT Policy #ISD-16-001 for its project executive to verify that the project team prepares all of the required project management plans. This formal process also includes a process to periodically verify the project team adheres to all plans. The policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #3 To: Veterans Affairs, Department of

To ensure accountability and independence between the provision of IPO and IV&V services on future IT projects, by September 2016 CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by the Technology Department.

60-Day Agency Response

CalVet has developed CalVet IT Policy #ISD-16-001 to ensure IPO and IV&V services on future IT projects use separate contractors/providers for each service. The policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #4 To: Veterans Affairs, Department of

To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016 CalVet should establish a process to periodically verify that its staff follow state contracting requirements and maintain all required contract documentation.

1-Year Agency Response

CalVet has revised it's IT Project Policy to include ensuring that its staff follow state contracting requirements and maintain all contract documentation.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

CalVet has added the contracting verification process to the CalVet Contract Management Handbook. Mandatory training for CalVet procurement and contracting staff was held on August 23 - August 25, 2016. The CalVet Contract Management Handbook and process was introduced and addressed at the workshop.

California State Auditor's Assessment of 6-Month Status: Pending

Although CalVet's response indicates that it added a contract verification process to its contract management handbook, this change does not establish a process to periodically verify that staff follow the requirements and maintain all required contract documentation. When we followed up with CalVet, staff stated that CalVet plans to draft a policy to establish a process to periodically verify staff maintain required documentation by the end of January 2017.

60-Day Agency Response

CalVet has added the contracting verification process to the CalVet Contract Management Handbook. Mandatory training for department procurement and contracting staff is scheduled for August 23 - August 25, 2016. The CalVet Contract Management Handbook and process will be introduced and addressed at the workshop. Additionally, the contracting verification process will be added to the CalVet Administrative Manual.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #5 To: Veterans Affairs, Department of

To ensure it maintains all documentation related to its IT contracts, CalVet should, by September 2016, establish a process to verify that all divisions comply with its policy requiring each division to submit a records retention schedule to its Office of Procurement and Contracting.

6-Month Agency Response

CalVet has revised its Administrative Manual to reflect a change in responsibility from the Office of Procurementand Contracts to the Administrative Services Operational Projects and Support Section (AdminOPS). In addition, CalVet has included a formal process to ensure each division submits an updated Records Inventory and Records

Retention Schedule to AdminOPS annually.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

CalVet has revised its Administrative Manual to reflect a change in responsibility from the Office of Procurement and Contracts to the Administrative Services Operational Projects and Support Section (AdminOPS). In addition, CalVet has included a formal process in to ensure each division submits an updated Records Inventory and Records Retention Schedule to AdminOPS by September 30 of each year.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: Veterans Affairs, Department of

To ensure that it only accepts deliverables and pays for deliverables that are complete and that meet the contract requirements, by September 2016 CalVet should establish processes to ensure that the project executive verifies that individuals assigned to project roles are adequately qualified and experienced.

6-Month Agency Response

CalVet has revised IT Policy # ISD-16-001 to establish a formal process, through IT Policy # ISD-16-002, for its project executive to verify that individuals assigned to project roles are adequately qualified and experienced. The policy has been disseminated to all staff.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

CalVet has established a formal process through CalVet IT Policy #ISD-16-001 for its project executive to verify that individuals assigned to project roles are adequately qualified and experienced. The policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Pending

The policy that CalVet refers to does not include specific processes for the project executive to verify that individuals assigned to the project are adequately qualified and experienced for their roles.

Recommendation #7 To: Veterans Affairs, Department of

To ensure that it only accepts deliverables and pays for deliverables that are complete and that meet the contract requirements, by September 2016 CalVet should establish processes to verify and maintain documentation of receipt of all contract deliverables before approving payment.

6-Month Agency Response

The process to verify and maintain documentation of all contract deliverables before approving payment is included in the CalVet Contract Management Handbook. The mandatory training for CalVet procurement and contracting staff was held on August 23 - August 25, 2016 and covered the responsibility of a contract manager.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

CalVet has confirmed that the process to verify and maintain documentation of all contract deliverables before approving payment is included in the CalVet Contract Management Handbook. The mandatory training for Procurement and Contracts scheduled for August 23 - August 25, 2016 will cover the responsibility of a contract manager

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #8 To: Veterans Affairs, Department of

To ensure that it only accepts deliverables and pays for deliverables that are complete and that meet the contract requirements, by September 2016 CalVet should establish processes to strengthen its contract management on all future projects by requiring the project manager to sign off on invoices along with the contract manager before approving payment.

1-Year Agency Response

CalVet has revised it's IT Project Policy to add the project manager and contract manager responsibility of signing off on invoices before payment is approved.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

CalVet has added the invoice approval requirement to the CalVet Contract Management Handbook. Mandatory training for CalVet procurement and contracting staff was held on August 23 - August 25, 2016. The CalVet Contract Management Handbook and process was introduced and addressed at the workshop.

California State Auditor's Assessment of 6-Month Status: Pending

The added invoice approval requirement that CalVet refers to does not require both the project manager and contract manager to approve invoices before payment, as we recommended. When we followed up with CalVet, staff indicated it will draft a policy to establish this process and plans to finalize it by the end of January 2017.

60-Day Agency Response

CalVet has added the invoice approval requirement to the CalVet Contract Management Handbook. Mandatory training for department procurement and contracting staff is scheduled for August 23 - August 25, 2016. The CalVet Contract Management Handbook and process will be introduced and addressed at the workshop.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #9 To: Veterans Affairs, Department of

To ensure that it maximizes its opportunity to successfully implement future IT projects, including its plan to replace its current system, CalVet should, by September 2016, establish a formal process to document the changes it makes as a result of the lessons-learned sessions it conducts.

6-Month Agency Response

CalVet has revised IT Policy # ISD-16-001 to establish a formal process, through IT Policy # ISD-16-002, to document the changes made as a result of lessons learned sessions for all future IT projects.

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

CalVet has established a formal process, through CalVet IT Policy #ISD-16-001, to document the changes made as a result of lessons learned sessions for all future IT projects.

California State Auditor's Assessment of 60-Day Status: Pending

Although CalVet refers to its new policy that requires its IT projects to capture and incorporate lessons learned for each project (or phase, if applicable), the policy does not address a process for documenting changes made as a result of the lessons learned.

Recommendation #10 To: Veterans Affairs, Department of

To ensure that it maximizes its opportunity to successfully implement future IT projects, including its plan to replace its current system, CalVet should, by September 2016, establish a formal process to verify that its staff conducts lessons-learned sessions for all key phases of the next project.

60-Day Agency Response

CalVet has established a formal process, through CalVet IT Policy #ISD-16-001, to verify that its staff conducts lessons learned sessions for all key phases of future IT projects. The policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #11 To: Veterans Affairs, Department of

To ensure that it maximizes its opportunity to successfully implement future IT projects, including its plan to replace its current system, CalVet should, by September 2016, establish a formal process to incorporate the recommendations identified in its Post Implementation Evaluation Report.

60-Day Agency Response

CalVet has established a formal process, through CalVet IT Policy #ISD-16-001, to incorporate the recommendations identified in its post implementation evaluation reports. The policy has been disseminated to all staff.

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #12 To: Technology, California Department of

To ensure that it can demonstrate that it is acting in the best interest of the State, the Technology Department should, by December 2016, create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies' contract disputes and negotiations for the termination of a contract and maintain those documents according to its records retention schedule.

1-Year Agency Response

This item is fully implemented. In March 2017, the CSA was sent CDT's internal procedures for documenting the processes associated with contract disputes or contract termination negotiations

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Planned Action this recommendation has not been completed due to other high priority workload demands. However, development of the internal procedures are underway, and the CDT needs to extend the planned completion date that was formerly December 2016 to March 2017.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The Department of Technology (CDT) agrees with this recommendation and will develop internal procedures for documenting the formal process associated with contract disputes or contract termination negotiations. Procedures include retaining a summary of negotiation outcomes maintained with applicable law and the CDT's records retention schedule.

Corrective Action Progress: The CDT internal procedures for the aforementioned Corrective Action Plan are currently under development.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #13 To: Technology, California Department of

To ensure accountability and independence between the provision of IPO and IV&V services, the Technology Department should, by December 2016, establish a written policy requiring departments that request and receive approval to contract for IPO services to use a different contractor than the one providing IV&V services.

1-Year Agency Response

The CDT updated State Administrative Manual, (SAM), section 4940.3 with language to address this recommendation.

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The CDT plans to incorporate the recommendation #13 changes into SIMM 45 to be among additional policy guidance concerning standardization of IV&V contracting terms. The final draft policy review is expected to occur in January 2017, with a new planned completion date that was formerly December 2016 to February 2017.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The CDT agrees with this recommendation and will develop policy so that if CDT allows state entities to use contracted vendors for Independent Project Oversight services, it will be under the management of the CDT and will not be the same vendor that provides Independent Verification & Validation (IV&V) services on that same project.

Corrective Action Progress: The CDT's policy for the aforementioned Corrective Action Plan is currently under development.

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #14 To: Technology, California Department of

Although the Technology Department indicated that its intent is not to outsource its statutory responsibility for IPO, in any instances where its staff conduct a portfolio review of a project's IPO, the Technology Department should, by December 2016, establish a process for its review of documents created by the agency's IPO contractor that includes verifying whether these reports include critical analysis of project progress and vendor performance so it can intervene when necessary.

Annual Follow-Up Agency Response From October 2019

The CDT Office of Statewide Project Delivery has developed written instructions to the project approvals and oversight staff using existing processes that provide the necessary controls and reporting functions to respond and address this finding and prevent the type outcome described in this report.

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2018

As indicated earlier, the Department of Technology (CDT) has no intent to outsource statutory IPO responsibility. In the unlikelihood that such an event were to occur, the CDT would exercise its authority under SAM 4940 and review third party IPO oversight documents, IPORs, PSRs, and project artifacts as guided by SIMM 45 (updated December 2017) and the project oversight review processes and methodology.

Within SIMM 45 existing and updated oversight instructions and guidance establish methods for determining and identifying variances from project cost, schedule and scope via the independent oversight project reporting (IPOR) and the Project Status Report (PSR). If a 3rd party IPO were operating in this instance, the CDT would conduct an assessment if CDT:

- Does not receive the required monthly IPO reports timely and would result in immediate escalation; or

- Receives those monthly reports and determine if they are non-compliant in form or substance with state IT policy; or

- Receives those monthly reports and if they indicate variances are unmitigated - CDT would initiate an escalation processes to address the concerns based upon overall risk to the project

California State Auditor's Assessment of Annual Follow-Up Status: Pending

The Technology Department did not provide documentation substantiating its claim of full implementation. Although its response states it would conduct an assessment of a 3rd party IPO if it outsourced this responsibility, it did not provide documentation demonstrating it has established a process for its review of documents created by an agency's IPO contractor.

1-Year Agency Response

The CDT has included language in both the State Administrative Manual (SAM)section 4940 and the Statewide Information Management Manual, section 45 to address this recommendation.

California State Auditor's Assessment of 1-Year Status: Pending

The language included in SAM 4940 states that project oversight services contracted by agency/state entities must comply with the IT Project Oversight Framework (see Statewide Information Management Manual section 45), and the requirements outlined in SAM 4910. SAM 4940 also states that the Department of Technology may conduct assessments to ensure compliance with minimum requirements. However, this section does not identify under what conditions it would do so. As a result, its process for reviewing documents created by the agency's IPO contractor, including verifying whether these reports include critical analysis of project progress and vendor performance, is incomplete. The Technology Department has indicated to us that it is revising its internal policies and expects to formally adopt them by the end of September 2017. We will reevaluate at that time.

6-Month Agency Response

The Planned Action for recommendation #14 has not been completed due to other high priority workload demands. However, development of the internal procedures are underway, and the CDT needs to extend the planned completion date that was formerly December 2016 to February 2017.

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The CDT agrees with this recommendation. The CDT will develop internal procedures for the IPO Division staff to evaluate vendor and state entity reports, as well as any additional input that is warranted given the condition, stage and trends of a project and to document the results. The procedures will reference the IPO Division escalation process in the event such action is warranted.

Corrective Action Progress: The CDT internal procedures for the aforementioned Corrective Action Plan are currently under development.

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2015-121

Agency responses received are posted verbatim.