California State Auditor Report Number : 2015-121

California Department of Veterans Affairs
The State Paid Nearly $28 Million for a Flawed System That Fails to Meet the Needs of Its Veterans Homes

Audit Results

The Enterprise-Wide Veterans Home Information System Does Not Meet the Needs of the Veterans Homes

The California Department of Veterans Affairs (CalVet) implemented the Enterprise-Wide Veterans Home Information System (system), but the system does not meet the original primary goals of the project. In its feasibility study report (FSR) requesting approval for the system, CalVet described the need for an information system that would improve the quality of care delivered to veterans by making its clinical documentation process more efficient; reducing reliance on paper records; providing a consistent, safe, integrated system of care; and improving regulatory compliance. However, the system implemented did not achieve those improvements.

The System Has Not Achieved Key Project Objectives of Improving Efficiency and Improving Quality of Care to Veterans

The system implemented has not improved the efficiency of the homes’ clinical documentation process or reduced their reliance on paper records because of flaws that staff encountered with the system. The administrators from four of the veterans homes, which use a majority of the system’s modules, including ADL Clinical—a main module of the system that the homes use to manage electronic health records—identified shortcomings with how the system works and indicated that the system is not user-friendly or intuitive. Specifically, some of the administrators told us that the system is difficult to navigate and results in staff spending unnecessarily long amounts of time locating or entering information multiple times into residents’ patient care records.

The chief medical officer at the home in Chula Vista told us that because of concerns staff had about the functionality of the system, given the numerous problems they experienced while using it, he had the staff conduct time studies of some routine tasks in July and early August 2013. Specifically, in a document he provided to CalVet headquarters, he described that staff in four departments performed five different routine tasks and the results revealed that these tasks required approximately twice as much time on average after the system was implemented. For example, a registered nurse determined that it took 105 minutes to conduct a new resident’s initial assessment using the system, compared to the 30 minutes for the same task before the system was implemented. In another example, a pharmacist found that it took seven minutes to process a medication order using the new system, a task that required 3.5 minutes before the system was implemented. Despite these concerns, in late August 2013 CalVet’s then-undersecretary of Veterans Homes responded that the time it was taking staff to complete processes with the system would likely decrease over time as they became more comfortable with the system and refined the business processes.

In addition to the extra time it was taking them to complete tasks using the system, staff found that they had to enter the same resident information into multiple modules of the system to provide care to the residents. The administrator at one home stated, for example, that when a home receives a new resident, staff must enter the resident’s information into the system’s admission module before they can admit the resident. A nurse then has to reenter that same information into the system’s assessment module before staff can conduct the assessment of the resident. Staff also reported that some system functions would regularly freeze or lock up during use. In one extreme instance, the entire system was down for more than a week at the end of August and beginning of September 2013.

Because of continuing concerns, in mid-2014 CalVet conducted an assessment of the system to determine if it was functioning properly and if it met the quality and needs of CalVet. In its response to CalVet’s assessment, the system contractor indicated that a majority of the more than 500 items identified required setup and training, and many other items could be resolved through enhancements. Further, the system contractor stated there were a dozen identified bugs or issues and that it needed clarification on about 100 items. CalVet’s final project manager stated that the system problems the homes experienced—for example, system slowness and freeze-ups—were a result of the system’s poor architecture and configuration.
Administrators of the homes indicated that at times, they resorted to manual processes to manage patient care because of the problems they encountered with the system. Two of the five homes that use the system’s ADL Clinical functionality—Barstow and Ventura—indicated that staff reverted back to paper processes when the system freezes or crashes. In fact, the management of the Ventura home stated that staff use paper documentation as a backup because they do not trust the system to work properly. Two other homes that use the system’s ADL Clinical module—the Chula Vista and Lancaster homes—also indicated that staff use paper documentation in addition to the electronic system for assurance. For the remaining three homes—those that use only a minimal number of the ADL Clinical module’s components—two stated that they continue to use paper records for clinical documentation while the third home continues to use the old system—Meditech—and other workarounds.

When we asked CalVet about how these functional problems have affected the quality of care veterans receive, its deputy secretary of Veterans Homes stated that the quality of care has not changed and that the homes continue to pass surveys performed by the United States Department of Veterans Affairs and the California Department of Public Health with the system in place. However, he added that the system is not maximizing efficiency because nurses are having to spend more time entering information in the system, limiting the time they are able to spend with residents. Therefore, we believe the overall quality of care provided to the veterans is negatively affected by these inefficiencies because of the increase in time that staff spend using the system.

Additionally, staff have encountered serious problems while using the order entry component—the system component that staff use to place a resident’s medication order—that have created a risk of harm to the residents. Specifically, staff reported problems that included the system recording incorrect medication dosages on prescriptions, failing to notify staff that a resident was allergic to a medication, and failing to note verification of whether the pharmacy had received prescription orders, resulting in the system not placing medication orders. For example, in March 2014, staff in one home placed on hold a resident’s medication to lower her cholesterol, and the system automatically expired the medication order after seven days because staff did not release the hold. Staff reported to the chief medical officer that the system did not alert the nursing staff that the medication order had expired, and as a result, the resident did not receive her medication for five months. In another example, the administrator at the Ventura home stated that although a veteran’s allergies are recorded in one module of the system, the system does not alert the nurse if a medication to which a resident is allergic is entered in the medication order module. In August 2014, staff at another home reported that a resident’s order for eye drops to treat glaucoma dropped from the system and the patient did not receive the drops for two months; the staff at the home were not aware that the resident was missing the eye drop medication until his son notified them. In response to these types of problems, CalVet suspended the use of the medication order entry module of the system in September 2014.

As CalVet identified during its assessment of the system in 2014, staff have to enter patient information more than once and they have encountered various problems in the medication order entry functionality. According to the final project manager, these problems occurred because the system contractor did not configure the system correctly. In April 2015, ADL—the software provider—also communicated to CalVet that based on its work to resolve outstanding trouble tickets over the past three months, it had concerns with the system’s configuration during installation and with the instruction of CalVet staff on the proper setup and implementation of the software. Our information technology (IT) expert reviewed the system’s draft configuration document and determined that it was incomplete and could not be reviewed for quality or correctness. He also noted that the draft architecture design document was incomplete and insufficient and would likely be a source of implementation problems. However, as we describe later in this report, it was CalVet’s responsibility to monitor its system contractor’s performance and to ensure that the system contractor provided deliverables consistent with the terms of the contract.

The Veterans Homes’ Use of the System’s Modules Varies

The system has also failed to serve as the integrated system of care that CalVet had planned and expected. CalVet envisioned its integrated system of care as enabling any veterans home to operate seamlessly with any other home, providing access to care information should a veteran move between homes. Because of the problems with the system, however, CalVet implemented only a portion of the system at the last three homes to receive it. According to the final project manager, headquarters and all of the homes received all modules of the system software but some did not receive training on all modules. He stated that when some of the planned tasks, such as training, are not completed, the module is not considered fully implemented. Specifically, three homes—Fresno, Redding, and Yountville—confirmed that although CalVet installed the ADL Clinical module, their staff either did not receive training on it or received training on only a minimal number of the module’s components. Additionally, according to CalVet’s deputy secretary of Veterans Homes, because of the problems the homes have encountered with the system, CalVet allowed individual homes to use the system’s modules in ways that worked best for each one.

CalVet’s decision to halt the training and limit the use of a system module that was not fully working and to allow each home to use the modules that worked best for that particular home is understandable given the significance of the problems the homes were experiencing. However, because CalVet did not overcome the problems with the system’s functionality, it failed to achieve one of its primary goals: a fully integrated system of care in which the homes’ use of the system is standardized and veterans receive consistent care. For example, as we discussed in the Introduction, CalVet’s FSR stated that under the previous system, when a resident transferred to another home, the staff at the new facility were unable to access the individual’s medical history electronically, hampering their ability to provide optimal care and increasing costs. In addition, the FSR indicated that increasing the amount of clinical documentation that the homes recorded electronically would improve CalVet’s ability to evaluate the outcome of services and procedures. It also indicated that improved cost reporting would enable CalVet to monitor and manage the costs of services across different facilities. However, because the homes’ use of the system’s modules varies and because the ADL Clinical module was not fully implemented at all of the homes, the staff at the homes cannot communicate with each other’s systems to share any necessary resident information. Table 3 shows when CalVet implemented the system at its headquarters and each of the eight homes as well as the extent of the system modules in use at each of those locations. Without a fully integrated system that is used in a standardized way, the homes cannot operate seamlessly with other homes and electronically share care information as the planned integrated system envisioned.

Table 3
Enterprise-Wide Veterans Home Information System Module Use by Location and Date of Implementation

	Headquarters, May 2012		Barstow, May 2012	Chula Vista, June 2013	Fresno, November 2013	Lancaster, May 2013	Redding, November 2013	Ventura, May 2012	West Los Angeles, May 2012	Yountville, November 2013
Multiple Component  Modules
ADL Clinical  (23 components) Software designed to address critical functions of providing long-term care by capturing and managing patient information.
Fully (F)	1		14	9	1	11	2	13	11	3
Partially (P)	0		5	5	0	4	0	3	6	1
Not used (N)	2		3	8	0	3	0	2	5	0
Not Implemented (NI)	0		0	0	22	0	21	0	0	19
Not applicable (NA)	20		1	1	0	5	0	5	1	0
ADL Financial  (36 components) Software used to support third-party billing, Medi-Cal, Medicare, accounts receivable, transaction history, withdrawals and charges; making, tracking, and controlling financial events.
Fully	12		30	29	23	13	24	26	21	29
Partially	7		3	4	5	11	5	2	7	2
Not used	1		3	3	6	10	5	6	8	2
Not Implemented	0		0	0	2	0	2	0	0	3
Not applicable	16		0	0	0	2	0	2	0	0
Single Component  Modules
Dynamics Financial accounting and business management software that automates creation and management of accounting data and workflow. Used to augment ADL Financial to support financial management. The specific component used is for purchasing and inventory.
	N		F	P	F	F	F	F	F	F
Gerimenu Software used for resident meal planning to help facilities run their nutrition departments more efficiently.
	NA		P	P	F	N	NI	N	F	P
Documentum An enterprise content management system used to house and  manage  electronic documents.
	N		F	N	NI	N	NI	F	F	NI
Framework LTC Pharmacy management software designed for long-term care and institutional facilities.
	NA		F	F	NI	N	NI	F	F	F

CalVet’s Assessment Results Overstated System Compliance Concerns

In a June 2015 response to questions it received from the Legislature, CalVet included the results of its assessment of the system and identified several areas in which its system was not complying with the provisions of certain federal and state laws and regulations. However, we found that CalVet is compliant with the laws and regulations it identified or, in one instance, can take steps to avoid reductions to federal reimbursements.

Although CalVet’s assessment results indicated that the system does not comply with the federal Health Insurance Portability and Accountability Act (HIPAA) regulations related to its ability to audit record activity, we found that it is compliant. The HIPAA privacy and security regulations require covered entities to implement appropriate administrative safeguards to ensure the protection of electronic personal health information. The regulations require, among other things, the implementation of audit controls, such as audit logs that record and examine activity in information systems that contain or use protected electronic health information. CalVet indicated that the system’s limited ability to audit health record activity makes it noncompliant with HIPAA. Specifically, according to CalVet’s agency information security officer, rather than having the ability to access audit logs directly from its system, CalVet must request audit logs from the software provider. Additionally, CalVet’s chief information officer stated that based on her experience, the time it takes the software provider to produce the logs for CalVet is unreasonable. However, the assistant director of the State of California Office of Health Information Integrity—which has statutory oversight of state entities’ HIPAA compliance—stated that as long as CalVet can request and receive audit logs, it is compliant with that requirement of HIPAA.

Additionally, CalVet indicated that the system does not comply with federal and state laws that prohibit the use of color coding in federal and state agency information systems as the only means of conveying information. For example, a form that requires a user to complete only the areas in red would be noncompliant. CalVet identified a concern that critical on-screen indicators within the system are coded based on color, and these indicators do not work for individuals who are color blind. However, in each example CalVet provided to us from the system, we found that the color coding was in addition to another means of conveying the information. For example, a patient’s discharge date is listed in red to indicate that the patient has been discharged; however, the existence of a discharge date would notify a staff member that the patient has been discharged; thus, the system does not rely solely on color coding to convey that information. As a result, CalVet could not demonstrate that the system inappropriately relied on color coding.

Further, a key provision of the American Recovery and Reinvestment Act of 2009 (ARRA) went into effect in January 2014, requiring public and private health care providers and other eligible professionals to adopt and demonstrate meaningful use of electronic medical records to maintain existing Medicaid and Medicare reimbursement levels. As part of ARRA, Congress mandated a reduction to Medicare Part B reimbursements for eligible professionals’ services if the eligible professionals performing the service did not demonstrate meaningful use of certified electronic health records. CalVet indicated in its assessment results that ADL—the system software vendor—is not certified. To determine the extent of these reductions in reimbursements for eligible professionals, we asked CalVet how many of its professionals were affected by the reductions. The staff services manager of CalVet’s Medical Cost Recovery and Support Unit indicated that although her staff reviewed accounting records and identified some eligible professionals who received the Medicare reimbursement reductions in 2016, the staff have not reviewed all facilities’ accounting records to determine all of the individuals affected. As of early May 2016, CalVet did not know how many of the homes’ eligible professionals will be impacted by the Medicare reimbursement reductions. However, in our review we noted that eligible professionals can apply for hardship exceptions through the federal Centers for Medicare and Medicaid Services in certain categories to avoid this payment reduction. One of the exemption categories is extreme or uncontrollable circumstances and includes, as an example, that the eligible professional’s electronic health record vendor was unable to obtain certification. Therefore, CalVet’s eligible professionals can apply for a hardship exemption as an option to avoid the reductions until ADL receives its electronic health records certification.

Although it is possible for CalVet to take reasonable steps to remain compliant with these laws and regulations, the need for such additional work contradicts the intent of the new system—to improve its operational efficiency. CalVet also intended for the system to improve its regulatory compliance; thus, the need for additional steps, as reasonable as they may be, indicates the system has not improved regulatory compliance.

Project Management Failed to Promptly Recognize the Severity of System Problems

In its governance plan for the project to implement the system, CalVet outlined the governance bodies that steered, controlled, and managed the project. The plan includes descriptions of the two main bodies—the project management team (management team) and the executive steering committee (steering committee)—responsible for overseeing the project, their roles and responsibilities, and the structure of each. The management team, led by the project manager (a role filled by CalVet’s deputy secretary for Veterans Homes Information Management), was responsible for managing the day-to-day operations of the project—such as the scope, schedule, and resources for the project—to ensure that the project achieved outcomes as planned. The steering committee was an advisory body that supported and provided perspective to the project executive (a role filled by CalVet’s undersecretary of Veterans Homes) that held total decision-making authority. According to the governance plan, the steering committee was to meet monthly to discuss major issues and risks related to the project and to make decisions. The steering committee was not directly responsible for managing project activities, but it was supposed to provide support and guidance to those who did. The text box shows selected key roles and responsibilities as described in the project’s governance plan.

As early as June 2012, staff began reporting problems with the system’s pharmacy component to the project executive and the former project manager, but they did not address the problems. The steering committee meeting minutes show that staff began reporting these problems in June 2012 and continued to note problems with pharmacy implementation in the monthly meetings through the beginning of 2013. Specifically, in July 2012, the pharmacist implementation coordinator noted that the pharmacy team had developed a problem log and would meet weekly to discuss concerns and attempt to identify a resolution. Again, in the August and October 2012 meetings, the pharmacist implementation coordinator noted the pharmacy team was identifying and tracking problems. The minutes from the October 2012 steering committee meeting also showed that the project executive had noted that the system contractor had done a good job on the pharmacy problems and that it had arranged regular meetings with software vendors to address problems. However, in November 2012 the pharmacist implementation coordinator noted that the pharmacy component was still experiencing problems receiving orders from the order entry component in ADL Clinical. For example, in January and February 2013, he noted that one home was experiencing medication orders missing from the system and as a result, the system was not generating some orders for the pharmacy to fill.

Given the severity and recurrence of the problems being reported—and the fact that they were clearly communicated in the steering committee minutes—we expected the meeting minutes to indicate that the project executive or project manager had discussed resolutions or actions to be taken to resolve the problems. However, in the minutes from February 2013, the project executive noted that the home administrators were keeping her updated on how they felt about the system and that it seemed many problems were being resolved through direct communication with the project team even though correspondence between one of the homes and headquarters showed otherwise as the problems continued. Specifically, correspondence to CalVet from the chief medical officer at the Chula Vista home in August 2013 indicated that significant problems with the pharmacy component remained and were increasing the risk of medication error. The chief medical officer noted that no verification was occurring that all medication orders were successfully transmitted, and some orders were not processed and fell into an error queue without notifying staff that the order had not been processed. In fact, CalVet eventually suspended the use of the medication order component at all eight homes.

Because that project executive and the project manager during that time are no longer at CalVet, we were unable to obtain their perspectives as to why they did not address the concerns with the pharmacy component raised in the steering committee meetings. According to the manager of the project management office (PMO manager) in CalVet’s Information Services Division (ISD), who was present at the time, the steering committee meetings were only status updates to the committee and resolutions were never discussed. Further, she stated that management originally thought the problems with the order entry functionality were mainly a result of user error and that staff needed more training.

CalVet’s project management did not begin to take steps to address reported problems until late 2013, having spent nearly $6 million from the time staff began reporting problems to the steering committee in mid-2012. According to the final project manager, when he stepped into his role in November 2013, he learned of numerous problems with the project from CalVet leadership and staff and from his review of project communications, documents, and invoices. In the same month, a staff member from CalVet’s Veterans Homes Division visited the five homes that had implemented the system’s ADL Clinical module—Barstow, Chula Vista, Lancaster, Ventura, and West Los Angeles—and noted the functionality problems the staff were reporting. Notes from those visits indicate that staff were experiencing several problems, including system modules that would freeze, information showing correctly in the system but appearing inaccurate when a report was generated, and their need to use paper documents or other workarounds as backup or in lieu of the system. According to the final project manager, he, the agency information officer, and the former deputy secretary of administrative services reported critical concerns to the California Department of Technology (Technology Department) in December 2013. CalVet then decided to pause the project to resolve critical system problems. The Technology Department’s director at the time indicated that the Technology Department directed CalVet to follow the dispute resolution process outlined in the contract with the system contractor, which was to issue a cure letter, a formal written request to provide specific deliverable documents and meet other requirements according to the terms of its contract. In January 2014, CalVet sent the system contractor a cure letter requesting the documents for the system architecture and system configuration deliverables, which provide the organizational structure of the system and the arrangement of a computer system’s components. The letter also required the system contractor to meet the identified requirements by February 13, 2014.

In early 2014, after the system contractor disputed CalVet’s claims in the cure letter, the Technology Department began facilitating discussions between CalVet and the system contractor. CalVet’s post-implementation evaluation report (PIER) stated that the Technology Department facilitated these discussions from February through March 2014.³ Although Technology Department staff told us it facilitated a series of meetings to try and resolve the dispute between CalVet and the system contractor, the Technology Department could only provide one document summarizing outstanding problems as of January 28, 2014. That summary identifies CalVet’s and the system contractor’s position on 16 problems. The summary also provides the Technology Department’s comments on most of the problems. Although the document provides a summary of the disputed issues, it does not include any specific action plans to resolve differences or any final resolutions.

Under the direction of the Technology Department, CalVet conducted an assessment of the system from May to August 2014 to determine if the system was functioning properly and was meeting its needs. The Technology Department’s former director told us that it required CalVet to conduct a series of assessments to figure out what was causing the problems. He also stated that the Technology Department lent a staff member—an enterprise architect—to CalVet to perform the assessment. However, the only documentation that the Technology Department provided from its enterprise architect’s review of the system is a one-page summary that lacks details of his assessment and a conclusion as to whether the system design was functioning properly. CalVet staff at the homes continued to experience serious functionality problems with the pharmacy component, such as errors in medication orders. As a result, in September 2014—just over a year and a half after it should have reasonably recognized the severity of the problems—CalVet suspended the use of that component of the system in all of the homes.

In December 2014, after unsuccessful negotiations, CalVet, the Technology Department, and the system contractor signed a settlement agreement to terminate the contract. We asked the Technology Department whether it performed an assessment to ensure that the settlement agreement was in the State’s best interest. The former director stated that the Technology Department performed an assessment of the deliverables listed in the cure letter and found that the system contractor had submitted some of the deliverables but identified others that it had not submitted, which were used as part of the settlement with the system contractor. However, the Technology Department was unable to provide us with documentation to support its claim that it conducted such an assessment of the deliverables to ensure that the settlement amount was in the State’s best interest. The former Technology Department director stated that a negotiator in its Statewide Technology Procurement Division (technology procurement division) negotiated the settlement agreement and that the negotiations were meant to determine what CalVet needed and what the system contractor was owed. Although we requested documentation from the Technology Department to demonstrate its efforts during the negotiations to ensure that the agreement reached was in the State’s best interest, it did not provide such documentation. According to the deputy director of the technology procurement division, the individual who led the negotiations and his manager have left state service and no documents on the negotiations are in the file they left behind.

CalVet’s documentation indicates that the system contractor had outstanding invoices to CalVet for over $1.9 million for full implementation and maintenance of the system in the homes in Fresno, Redding, and Yountville as well as for the remaining deliverables it indicated it had already completed. However, the staff in those three homes did not receive training on and are not using the ADL Clinical module, a large portion of the system that the homes would have used to manage their electronic health records. As such, CalVet valued the portion of the system that the system contractor did implement and agreed to a lower settlement amount of $350,000. At that point, CalVet had spent $26.2 million implementing the system. In January 2015, CalVet and the Technology Department mutually agreed to close out the project.

As of January 2015, CalVet discontinued its efforts to address the functionality problems it had identified and decided to use the functionality available from the system to the best of its ability while it explored options for a replacement solution. According to minutes of the steering committee meeting, CalVet’s acting secretary gave direction in January 2015 to cease any work to restore the order entry functionality, to prepare the system’s PIER, and to initiate efforts to replace the system. As of November 2015, CalVet continues to use the functionality available from its system. According to the current deputy secretary of Veterans Homes, CalVet is in the process of looking for a replacement system.

Failure to Follow Project Management Oversight Plans Contributed to CalVet’s Unsuccessful Implementation of Its System

Because it did not establish and follow all of the key activities of the project management plan that the Technology Department’s California Project Management Methodology (project management methodology) requires, CalVet missed opportunities to detect deficiencies and take corrective actions earlier. The planning stage of that project management methodology describes the development of a project management plan that includes a series of 10 plans that serve as a customized, orchestrated project management workflow for IT projects. The project management methodology document states that the purpose of project management is to ensure that the delivered product, service, or result meets the customer’s requirements and is delivered on time and within budget. Further, it states that a project management methodology improves the quality of project planning, communication, and control of executive and closure processes and thus improves the quality of the deliverables. The completion of the project management plan is the main objective of the planning stage of the project management methodology, which CalVet specified that it would comply with in its FSR.

Although CalVet hired a consultant to assist it in creating the series of 10 plans that make up the required project management plan, it did not develop one of those plans and four others were incomplete. Further, CalVet did not complete one of two additional required plans separate from the series of project management plans and did not consistently follow the plans it did develop. Of the six complete plans, CalVet fully followed only four. Table 4 shows the 12 required plans and indicates which ones CalVet developed and followed.

 

Table 4
California Department of Veterans Affairs’ Adherence to Required Project Plans

	Purpose	Was the plan developed?	If developed, did CalVet follow the plan?
Project Management Plan
Scope Management Plan	Define and document what is and is not within the project boundaries, and ensure that the project includes all the work required, and only the work required, to complete the project successfully.	No	NA
Configuration Change Control Management Plan	Document, control, and manage changes to key project components and deliverables throughout the project life cycle.	Yes	No
Human Resources Management Plan	Identify how and when labor needs will be met to ensure the project has sufficient staff with appropriate skill sets and experience. Additionally, identify and document project roles, responsibilities, and reporting relationships.	Partially	Partially
Communication Management Plan	Determine information and communications needs of project stakeholders: who they are, when they will need information, what information they will need, and how it will be given to them.	Partially	Partially
Risk Management Plan	Decide how to approach, plan, and execute risk management activities for the project. Establish agreed‑upon basis for evaluating risks and ensuring that sufficient resources and time are allocated for risk management activities.	Yes	Yes
Cost Management Plan	Plan, estimate, and control costs so that the project can be completed within the approved budget.	Yes	Yes
Quality Management Plan	Identify which quality standards are relevant to the project and determine how to satisfy them.	Yes	Partially
Schedule Management Plan	Provide for the timely completion of the project. The plan establishes how the project schedule will be managed and controlled and includes estimating the duration of activities.	Partially	Yes
Procurement Management Plan	Determine which project needs can best be met by purchasing products or services outside the project organization, and which project needs can be accomplished by the project team.	Yes	Yes
Contract Management Plan	Document the products, services, and results requirements needed to meet the project’s objectives.	Partially	No
Other Required Plans
Organizational Change	Assess stakeholders’ awareness and influence, determine any resistance or concerns, and identify optimum communication and actions to be taken.	Yes	Yes
Management Plan
Maintenance and Operations Transition Plan	Document how the project will be transitioned to the operational team that will own the new system. Also includes a bridge between the team executing development, the transition team, and the operations team.	No	NA

 

For example, CalVet did not develop a scope management plan, which defines and documents what is and is not within the project boundaries. That plan describes how a project’s scope is defined, verified, and controlled. If developed and followed, it helps a project’s decision makers determine whether the benefits of a proposed change in scope are worth the change in costs. This question needs to be answered every time someone on a project requests a change in scope. Because CalVet did not develop this plan, it did not have a process to ensure that proposed changes to the project included only the work required to complete the project successfully and to remain within the boundaries of the defined scope.

Additionally, as shown in Table 4, although CalVet had a complete quality management plan, it only partially followed that plan. The quality management plan identifies quality control activities as a key component of overall quality management, essential to ensuring a successful project. CalVet’s quality management plan stated that the project team would conduct structured reviews to determine whether project baselines were being maintained and to confirm the implementation of change requests and corrective actions, among other tasks. These reviews were to evaluate project progress against project objectives for quality, as defined in the SPR or project charter, and to determine whether shortfalls and causes of problems had been addressed, including compliance with organizational and project policies, processes, and procedures. CalVet’s PMO manager stated that CalVet never conducted these structured reviews. As a result, it did not recognize that it incorrectly reported project progress in its project status reports. As part of its oversight, in August 2012 the Technology Department identified that CalVet’s July 2012 project status report contained costs and milestones that were not aligned with the last approved governance document. As discussed later in this report, CalVet’s project status reports gave the impression that the project was still progressing according to the approved schedule when it was in reality nearly seven months behind. Had CalVet been conducting structured reviews to determine whether the project baseline was being maintained, it would have been able to identify this issue itself.

Further, CalVet did not follow its configuration change control management plan (change plan). This plan describes the process the project team will follow to document, control, and manage changes to key project components and deliverables throughout the project. For example, CalVet’s change plan specified that changes to technical or functional requirements, deliverables, or the project schedule were subject to the formal processes specified in the plan. Because CalVet did not follow the change plan, it was unable to demonstrate that it properly understood the impact of different change requests on the project’s costs, scope, and timeline. For example, one aspect of the change plan procedures that CalVet did not follow required the change control manager to assign an analyst to conduct an impact analysis of each change request the project team received. According to the change plan, the analyst was to complete an assessment of the proposed change and the change control manager would indicate whether the impact analysis affected the project’s scope, cost, schedule, or resources. The change plan also specified that the analyst would document the methodology followed and the steps taken to perform the analysis, clearly describing all assumptions and constraints. The change plan also stated that the analyst should summarize all final recommendations in a succinct fashion for subsequent review by the project manager, change control board, and project executive, including whether the change would have an impact on the project’s scope, cost, schedule, or resources.

Despite these requirements, CalVet made a significant change without following its formal change process. Specifically, a change request the system contractor submitted in September 2011 requested a modification to the conditions under which user acceptance testing (UAT) could proceed. UAT tests what the system contractor delivers to determine whether the deliverables conform to contract requirements. The change request asked to modify the language in the contract’s statement of work related to the commencement of UAT. The original language required CalVet to approve the system contractor’s certification of successful completion of system integration testing before proceeding to UAT, and the change requested that CalVet and the system contractor be allowed to approve acceptable progress in system integration testing before proceeding to UAT. The change request justification states that delays in the system environment readiness and in the development of certain modules would prevent CalVet from completing verification of all requirements before beginning UAT. A review of this change request indicates that it was approved by CalVet without completion of the impact analysis that its change plan required.

Because CalVet did not conduct an impact analysis for this change request, it cannot demonstrate that it properly considered the effects this proposed change would have on the project. As a result of this approved change request, CalVet began UAT before the system integration testing was completed, according to its project schedule. Therefore, CalVet could not test parts of the system for which development was delayed and it could not ensure that what the system contractor delivered conformed to contract requirements. As we described in an earlier section, CalVet’s final project manager indicated that some of the problems identified during its assessment of the system, such as the need to enter patient information more than once and problems using the medication order entry functionality, were the result of incorrect system configuration by the system contractor. Had an impact analysis been conducted for this change request, it would have considered the negative impacts of beginning UAT testing before system integration testing was completed, allowing CalVet’s project management to make an informed decision.

By not fully developing and following all of the plans that made up the required project management plan, CalVet limited its ability to detect deficiencies earlier in the development of the system. According to the final project manager, based on this review of project documents, the project management plans were not followed because oversight was lacking and because problems with the plans were not being escalated to the project executive. He stated that if project management plans were not being followed during the project, the individual assigned ownership for the plan would have been responsible to escalate the problem to the project manager. He further stated that if the project manager could not resolve the problem, it should have been elevated to the project executive, who could also move the problem on to the steering committee. CalVet’s governance plan states that the project executive provides oversight as needed and has responsibility to resolve project problems that cannot be resolved at lower levels. Our review of steering committee meeting minutes from the project repository did not show that problems regarding the plans were ever escalated to the project executive.

CalVet hired a contractor to perform both independent verification and validation (IV&V) services and independent project oversight (IPO) services for its system project. We refer to this contractor as the oversight contractor. CalVet’s contract management plan stated that the oversight contractor, in its IPO role, was responsible for monitoring the project plans and processes, assessing the project’s adherence to required project management processes and methodologies, and providing recommendations for improvement related to the project management effort. The final project manager stated that the oversight contractor should have independently been reporting on and identifying any nonadherence to project management plans and that this would have ensured that even if the problems were not being escalated up the chain of command, decision makers were being made aware of the problems. However, as we describe in the next section, the IPO reports the oversight contractor created did not identify the critical problems with the project. CalVet could not explain why the problems regarding adherence to the project management plan were not escalated to the project executive. Had the problems with the plans been so escalated or had the oversight contractor detected these problems, many of the problems experienced with the system project, such as incomplete project management plans, could have been addressed at an early stage.

Inadequate Independent Oversight of the System Project Left Stakeholders Without the Information Necessary to Ensure a Successful Implementation

The services that the oversight contractor provided were inadequate. Specifically, we noted missing critical deliverables we expected it would have provided through its contracted IV&V services. Additionally, our IT expert stated that in his opinion, the IPO was largely ineffective. As discussed in the Introduction, IV&V services provide a client with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and these services also provide an early warning of process and technical discrepancies and problems that might not otherwise be detected until late in the project life cycle. IPO services provide an independent review and analysis of project management practices to determine whether the project is being well managed.

CalVet’s oversight contractor provided both IPO and IV&V services. CalVet’s contract required the oversight contractor to perform tasks and activities in accordance with applicable Institute of Electrical and Electronic Engineers (IEEE) standards, which are industry standards also recommended in state policy. According to our IT expert, the relevant IEEE standard focusing on system, software, and hardware verification and validation processes does not require an organization that is performing traditional technical IV&V services to be organizationally independent from one that is performing optional project management oversight support.However, he stated that separation of IV&V and IPO duties is important and provides a number of advantages to the project and to the State. For example, IPO should assess whether appropriate IV&V services have been procured for the project. It is also within the purview of IPO activities to assess whether the IV&V contractor is doing its job well, such as whether IV&V services are effective and timely. If a contractor is performing both activities, there is also a potential conflict of interest because IPO might recommend that more of the IV&V services it is providing be used.

Our IT expert reviewed the IPO and IV&V reports the oversight contractor provided to CalVet and noted that he did not find evidence that the oversight contractor prepared certain key IV&V reports that the contract required. These reports should have provided insights into the technical quality of the work being performed. Some of the more critical missing IV&V deliverables included the following:

• Requirements traceability reports monitoring the tracing of project requirements throughout the project life cycle to ensure that the system meets specified requirements.
  
  
• System and software verification report assessing whether the work products satisfied the conditions established at the start of the development phase and fully addressed the requirements.
  
  
• System and acceptance report and testing report assessing whether the delivered products had been thoroughly tested to confirm that they were functional and met contractual requirements.
  

According to our IT expert, these IV&V reports are all critical. However, CalVet’s final project manager stated that the oversight contractor did not provide these deliverables to CalVet and that poor contract management by CalVet was the reason it did not identify the oversight contractor’s failure to submit the reports and require that it do so. By failing to obtain the technical assessments from these reports, CalVet neglected its responsibility to ensure that it was providing stakeholders with information about whether its project to implement a new system fulfilled all technical requirements and was functioning as expected.

In addition to missing critical IV&V deliverables, our IT expert concluded that the IPO was largely ineffective. He stated that IPO is there to make sure that the project manager is accurately reporting project status and that IV&V is providing independent technical analysis. His review of IPO and IV&V reports found almost no technical assessment; in addition, although the reports indicated what happened, they offered minimal analysis, with no explanation of root causes or trends. For example, one IV&V report referred to a checklist for delivered training materials indicating that there were problems, but it did not identify the problems and stated that the IV&V review could not be completed. In this case, according to our IT expert, IPO should have raised concerns that IV&V could not complete its work. Similarly, effective IPO should have raised concerns that IV&V was not managing requirements traceability, as mentioned earlier. Although the oversight contractor’s IPO reports should have identified these types of deficiencies in the IV&V work—performed by the same contractor—they did not. Because the IPO reports did not identify that IV&V was failing to perform critical work, neither CalVet nor the Technology Department received an accurate assessment of whether the processes the system contractor was using were effective and whether the system reflected the agreed-upon quality and solution.

The IPO reports also missed critical information about a variation from the approved project schedule, a variation later identified by the Technology Department. State policy requires that IPO reports assess the expected completion of tasks and milestones compared to the approved project schedule contained within the most recent SPR. However, in its August 2012 IPO report, the oversight contractor acknowledged that previous reports had looked at past project milestone dates and their actual completion dates rather than dates in the most recent approved SPR as it should have to determine whether the project was on schedule. Further, the IPO report stated that beginning with the August 2012 report, future tasks and milestones would be compared to the most recent project schedule approved in the SPR to determine whether the project was on schedule. Because the oversight contractor did not use the correct dates to measure project progress, the variation from the approved schedule was not identified, giving the impression that the project was still progressing according to the approved schedule when it was not.

CalVet’s final project manager, who identified the variance while he was employed at the Technology Department, informed us he met with CalVet’s former project manager and the oversight contractor to discuss the variance. As a result, the oversight contractor adjusted its tracking in the August 2012 IPO report to correctly reflect the schedule contained in the approved SPR. In that August IPO report, the oversight contractor reflected that the project was nearly seven months behind the approved schedule. However, if the oversight contractor had been using the correct dates to measure the project’s progress, it would have better informed project stakeholders on the true status of the project’s progress. Further, both CalVet and the project’s decision makers would have been aware earlier that the project was behind schedule and could have taken steps to address variances. CalVet’s final project manager could not explain why his predecessor or the oversight contractor did not use the approved schedule to measure performance, but in his opinion, it was likely that they were familiar with the requirements.

As we discussed earlier, our IT expert believes that by using the same contractor to perform both IPO and IV&V functions, CalVet did not ensure it had effective oversight for the project. The deputy director of the Technology Department’s IT Project Oversight Division stated that starting in July 2013, the Technology Department began assigning its own staff for IPO on reportable IT projects it deemed to be of medium or high complexity and that agencies would have to obtain explicit approval to contract for IPO services; this approval would only be granted if the Technology Department did not have enough staff to provide the services itself. She also stated that the Technology Department does not have a policy requiring agencies to obtain IPO and IV&V services from separate contractors; however, it is the department’s expectation that agencies receiving permission to use outside IPO services will obtain IV&V from separate contractors and she agreed this would be a good policy to formalize.

The Technology Department Did Not Adequately Fulfill Its Responsibilities in the Oversight of CalVet’s System

The Technology Department’s review of IPO reports for CalVet’s system project was inadequate, and as a result, it did not identify critical concerns and take timely action. The Technology Department received oversight authority after CalVet’s system project was already under way, and it made the decision to allow CalVet to continue with its contracted oversight rather than perform the IPO role itself. According to the deputy director of the Technology Department’s IT Oversight Division, because its oversight consisted of a review of project reports that CalVet’s oversight contractor was producing, it would have had no way of consistently identifying critical concerns related to the project if no significant problems were reported in these documents, as was the case with the CalVet project. However, our IT expert’s assessment of the IPO reports indicated that the fact that these reports did not contain critical information about the project or offer analysis of project progress or vendor performance should have triggered closer review and inspection.

State law transferred IT oversight authority from Finance to the Technology Department in January 2008.⁴ The Technology Department’s Statewide Information Management Manual defines project oversight as independent review and analysis of specific project activities and documentation to determine if the project is on track to be completed within the estimated schedule and cost and if it will provide the functionality the sponsoring entity requires. The oversight responsibilities include requiring agencies to provide the Technology Department with periodic reporting that describes the degree to which an IT project is within approved scope, cost, and schedule; project issues, risks, and corresponding mitigation efforts; and the current estimated schedule and costs for project completion.

According to the Technology Department’s former director, before December 2013, its primary role in the oversight of CalVet’s project was reviewing IPO and IV&V reports that CalVet’s oversight contractor prepared. The FSR for CalVet’s system, which was approved in 2007 by Finance—before state law transferred oversight responsibility for IT projects to the Technology Department—specified that CalVet would retain a contractor to perform IPO and IV&V for the overall project. In December 2007, CalVet entered into a contract, totaling just over $1.8 million, for these services. Although the former director added that he had regular verbal briefings—referred to as portfolio reviews—with both CalVet and the oversight contractor to discuss the status of the project, the frequency of these meetings could vary, ranging from monthly to quarterly. He told us no documentation exists from these meetings.

Although we previously described one instance in August 2012 when the Technology Department identified concerns with information in the IPO report and took action to address its concerns by meeting with the oversight contractor, overall we found that the Technology Department’s oversight of the IPO reports for CalVet’s system project was inadequate. Specifically, its oversight was not rigorous enough to detect concerns reported in the IPO reports or to question why the reports the oversight contractor prepared did not contain critical information about the project. As previously noted, our IT expert stated that the IPO reports the oversight contractor prepared did not offer an analysis of project progress or vendor performance and that should have triggered the Technology Department’s closer review and inspection. For example, the IPO report for February 2011 indicated that the project was using about half of the allocated resources for the fiscal year but did not offer an analysis of the cause or the implications this would have for the project. The same IPO report identified a risk that the system contractor might not be able to perform or meet the requirements according to its bid for the project; however, the IPO report did not include any significant analysis, thus missing an opportunity for IPO to provide an explanation of what it was seeing. In another example, our IT expert noted that in comparing the IPO reports for March 2011 and April 2011, he found a two-month delay in the projected end date for the design, configure and development phase; however, there was no comment explaining the sudden two-month slippage in the schedule.

Our IT expert also noted that in the IPO report for June 2011, the rating in “quality for architecture/system performance” was suddenly listed as inadequately defined even though in previous reports the notation was not applicable. He stated that this is a huge red flag, and he noted that the corresponding analysis said that the current draft of the system architecture document did not adequately and accurately reflect the planned system. However, no further analysis or context was provided about the impact of this change. Our IT expert stated that the lack of analysis and general comments should have been a red flag to anyone familiar with the IPO report format. He explained that there is a difference between tracking the project—what is happening, and oversight­—reporting what is happening and providing context for people to understand what is important, what is not, and the implication of what is being observed. He stated that CalVet’s project IPO reports all fell into the tracking category as they rarely offered analysis or perspective on what was happening; they simply reported schedule slips and risks. He added that any review of these IPO reports should have resulted in the Technology Department’s further inquiry of the oversight contractor and perhaps coaching to improve the quality of the reports. Although the IPO reports generally lacked this analysis of events and risks reported, we found no evidence that the Technology Department raised concerns about the quality of the reports with either the oversight contractor or with CalVet.

In fact, the Technology Department could not demonstrate that it had even reviewed IPO reports before August 2012, when, as discussed earlier, it raised an issue about the information in the oversight contractor’s report. At that point, CalVet had spent $15.8 million on the project. Additionally, the Technology Department could not locate any IV&V reports for CalVet’s project. Had the Technology Department’s oversight of the system been more rigorous, it should have identified and raised concerns about the deficiencies of the IPO reports and missing IV&V reports, which likely would have resulted in the oversight contractor informing CalVet earlier of problems with the system’s implementation.

The deputy director of the Technology Department’s IT Oversight Division stated that division staff were reviewing IPO and IV&V reports from CalVet’s oversight contractor, but because the reports were not indicating critical errors with the project, the Technology Department did not raise any concerns. However, we believe it should have. As we noted earlier, our IT expert stated that the lack of critical information about the project or analysis of project progress or system contractor performance in the IPO reports should have been sufficient cause for concern.

According to that deputy director, starting in July 2013 the Technology Department began assigning its own staff to perform project oversight. However, the former director told us that it devoted its limited staff resources to other troubled projects and only provided a portfolio review of CalVet’s relatively small system project when compared to those other troubled projects going on at the same time. Further, the deputy director indicated that because the Technology Department had an insufficient number of qualified staff to replace vendors on existing projects, CalVet was allowed to continue to use its oversight contractor. She stated that the Technology Department’s practice as of July 2013 has been that any IT projects deemed to be of medium or high complexity must obtain explicit approval from the Technology Department to contract for their own IPO services and approval is granted only if the Technology Department does not have enough staff to provide the IPO services itself.

In a previous audit report issued by our office, we identified concerns with the Technology Department’s limited resources and with its poor documentation of its oversight efforts on IT projects.⁵ The problems that we noted in that report were occurring at the time the Technology Department was providing oversight to CalVet’s implementation of its system. Specifically, our previous report noted that the Technology Department’s oversight and consulting division hired additional IPO analysts between fiscal years 2011–12 and 2013–14; however, we found it was unclear whether the division had enough positions at that time to effectively oversee the State’s IT projects. As a result, we recommended that the Technology Department conduct a workload assessment to determine the level of staffing and expertise required for the projects it oversees and using that workload assessment, it should make decisions to assign its staff to oversee each IT project. In March 2016, the Technology Department indicated in its one-year response to our earlier report that it had established processes to capture and evaluate workload information. If it follows these new processes, the Technology Department can better identify the level of oversight it can provide to state IT projects.

That previous audit report also noted that although the Technology Department is generally able to hire personnel to fill its IPO analyst positions, the job classification it uses may not attract applicants with the most relevant skills and experience required for IT project oversight. To ensure that it attracts and retains employees with appropriate experience and qualifications to perform IT project oversight, we recommended that the department continue its efforts to gain approval to use the project manager classification for its IPO analyst role. In its one-year response in March 2016, the Technology Department reported that it determined in October 2015 that the project manager classification was not suitable for use by the IT project oversight division. The Technology Department also indicated that the California Department of Human Resources (CalHR) is leading a significant effort to modernize a variety of state classifications, including IT classifications and the Technology Department is an active participant in CalHR’s reform initiative. Therefore, the Technology Department stated that until better options become available, it will continue to use the current classification—data processing manager—to fill its project oversight role. Nevertheless, we believe that by fully implementing the recommendations from our prior report and fully implementing the additional recommendations we present in this report, the Technology Department can ensure that it fulfills its responsibility of providing oversight to state IT projects.

CalVet Did Not Maintain Adequate Documentation for Its Selection of the System Contractor and for Some Key Deliverables

Because it did not maintain required documentation for selection of the contractor to implement its system, CalVet cannot demonstrate it complied with contracting requirements, and this lack of documentation raises questions about the prudence of its decisions. As of July 2010, the State Contracting Manual requires state entities to use an RFP for acquisitions of IT projects exceeding $1 million. State law in 2010 required agencies to award the contract to the bidder that achieves the highest score on its proposal. Regarding record retention, the State Contracting Manual states that departments are responsible for maintaining records in sufficient detail to allow anyone to review that documentation and understand how the procurement was requested, conducted, awarded, and administered. The State Contracting Manual further states that record retention varies depending on document type and can vary by department, depending on its internal retention schedule. Generally, these procurement documents should be retained for seven years from the end of the fiscal year in which the contract amount is liquidated—either through completion or termination.

CalVet could not provide some of the documentation it should have maintained related to its award of the contract for its system. CalVet’s contract records show that in June 2010 it received proposals from seven vendors in response to its RFP. However, according to its final evaluation and selection report (evaluation report), it reviewed six proposals and CalVet’s selection committee deemed five of them nonresponsive because of material deviations. For example, according to the evaluation report, the selection committee found that the documents submitted for one proposal did not substantiate that the project manager met the experience requirements outlined in the RFP. The evaluation report noted that other proposals did not provide supporting references or documentation verifying that the vendor could carry out the RFP requirements. In addition, one of the seven proposals included on the receipt log was not included in the evaluation report as having been evaluated. When we asked CalVet why it was not included, the PMO manager could not answer the question as she was not part of the procurement; further, she stated that there is no one remaining at CalVet that had worked on the procurement. CalVet’s final evaluation report indicated that CalVet scored and ultimately awarded the contract to the only responsive bidder. However, CalVet could not provide the proposals for six of the seven responding vendors, nor could it provide its evaluation documents for three proposals, including the winning proposal. When CalVet does not ensure that it maintains this documentation, it cannot demonstrate that it made a prudent decision or that it complied with state contracting requirements when selecting the vendor for its project.

CalVet could not adequately explain why it did not follow contracting requirements. The PMO manager agreed that poor record keeping by the previous contract manager contributed to the difficulty in locating the required contract documentation. Further, she stated that CalVet does not have a policy to periodically verify whether its staff are following contract documentation requirements. Although CalVet’s policy states that each division must provide a records retention schedule approval form annually to its Office of Procurement and Contracting (OPC), according to an OPC manager, OPC does not have a record retention schedule from CalVet’s ISD, the division responsible for CalVet’s IT contracts. Additionally, the final project manager could not provide a records retention schedule for ISD.

CalVet also could not demonstrate that it received sufficient documentation for some key system deliverables even though it approved payments for them. Specifically, CalVet’s project files indicate that it accepted and approved payments totaling $733,000 for three key deliverables—system design, system configuration, and UAT—even though it could not provide adequate documentation of receiving these final deliverables. The State Contracting Manual requires state agencies to verify that the goods and services they receive are satisfactory before approving payment for them. Further, CalVet’s contract for implementation of the system required that the system contractor submit all deliverables to CalVet for acceptance and then approval.

The system architecture design and configuration deliverables provide the organizational structure of the system and the arrangement of the computer system’s components. Although the contract manager at the time accepted the system configuration and architect design deliverables and approved the payment of $104,500, CalVet could not provide the final documentation. Instead, it could only provide us with a draft version of those documents. CalVet’s final project manager confirmed that no one has been able to provide the final documents for these two deliverables. He stated that many of the system problems the homes encountered were generally related to deficiencies in the system’s configuration, design, and training. For example, as discussed earlier, some homes indicated that staff have to enter the same resident information into multiple modules within the system, which the final project manager indicates is a result of poor system configuration. Communication between CalVet and executives at ADL—the system’s software provider—indicated that based on its work with CalVet to resolve outstanding trouble tickets over three months in early 2015, ADL was concerned with the system’s configuration during installation and with the instructions to CalVet staff on the proper setup and implementation of the software. In June 2014, in response to CalVet’s system assessment of functionality problems, the system contractor acknowledged there were more than 500 outstanding items of which about 240 would require further system configuration and additional training of subject matter experts. Because the contract manager at the time CalVet accepted the system design and configuration deliverables is also no longer at CalVet, it is unclear why she signed the acceptance letter and approved payment for those deliverables as CalVet does not have proof it received the final deliverables.

The UAT deliverable documents the testing that verifies that the system meets contract requirements and performs at a satisfactory level. For deliverables related to UAT, the contract required the system contractor, among other things, to develop a UAT report that included a description of the defects CalVet identified during UAT, the business processes and system functions or interfaces impacted by each defect that could not be resolved, and a corrective action plan for defects that could not be resolved. CalVet’s final project manager indicated that after he started at CalVet in November 2013, he looked for the UAT documentation but there were no tangible test results or tracking of specific requirements. He stated that for UAT, test cases are usually developed that include the test steps and the success criteria used to evaluate whether the given test case or scenario passed or failed. He added that there should be documentation for each test case completed by each tester.

Although the contract required specifics on the testing and results, the deliverable from the contractor for the UAT that CalVet provided to us included only a summary of the tests completed. That summary report did not include information about the specific requirements tested, who conducted the tests, the outcome of each test, the defects identified during the testing, or corrective action plans for each defect that could not be resolved. CalVet’s test manager confirmed that CalVet does not have the completed scripts that indicate who conducted the tests and what the outcomes were. As a result, CalVet cannot demonstrate that UAT was completed even though it approved payment of three invoices totaling $628,527 for UAT. In fact, CalVet’s oversight contractor reported in February 2014 that during UAT, CalVet had not accurately captured the results from the testers, and as a result, there was no empirical evidence to assess to determine whether the developed system met CalVet’s business needs. Because CalVet accepted deliverables and approved payment without sufficient documentation to demonstrate that the system contractor had satisfactorily completed them, CalVet failed in its contract management responsibilities to verify that the system it paid for was properly designed and configured, met contract requirements, and performed satisfactorily.

A number of factors contributed to CalVet’s failure to assess the adequacy of UAT deliverables. The test manager stated that because she did not have experience as a test manager in 2011, she could not determine whether the documentation the contractor submitted for testing was sufficient. She further explained that when she voiced her concern that she had never been a test manager to the project manager at the time, the previous contract manager, and the previous agency information officer, they told her it was okay and that they just needed someone to sign off on the testing. She further indicated that CalVet did not complete UAT before the system contractor implemented the system in the homes. Because CalVet did not complete UAT, it missed its opportunity to identify functionality issues before implementing the system in the homes. The test manager indicated that because UAT took longer than expected, the project team received outside pressure from the former project executive, CalVet’s former secretary, and the Technology Department to move forward with implementation to keep the project on schedule. Therefore, CalVet did not complete UAT before implementing the system.

Additionally, despite the lack of evidence for certain key IV&V deliverables, CalVet approved invoices from its oversight contractor for those deliverables. As we described earlier, our IT expert reviewed IV&V deliverables and did not find evidence that the oversight contractor prepared requirements traceability matrix reports—reports monitoring the tracing of project requirements throughout the project life cycle to ensure that the system meets specified contract requirements. In our review of invoices from the oversight contractor, we identified invoices totaling just over $12,000 for monitoring requirements traceability, but the contract records did not contain any evidence of project manager review or approval of those invoices. Further, CalVet was unable to provide us with documentation that its oversight contractor actually completed the review of requirements traceability. Therefore, it is unclear why the previous contract manager approved payment of the oversight contractor’s invoices. Additionally, according to a 2014 report, the oversight contractor stated that CalVet did not accurately capture the results from the UAT testers and as a result, there is no empirical evidence to assess whether the developed solution met the users’ business needs. The final project manager cited poor contract management as the reason why CalVet did not identify deficiencies and require the oversight contractor to fulfill the deliverables. However, he indicated that the IV&V contract CalVet signed with its oversight contractor in September 2014 now requires the oversight contractor to submit deliverable acceptance documents signed by the system’s project manager and contract manager along with invoices to ensure that only accepted deliverables are paid.

CalVet Identified Some Lessons Learned but Rarely Used Them to Improve Future Phases of Implementation

Although CalVet conducted lessons-learned sessions at points throughout the project from initial procurement through implementation at the Chula Vista home, it generally cannot demonstrate that it used those lessons learned to make improvements during later implementation phases. Additionally, CalVet did not conduct sessions to identify lessons learned during the final implementation phase at the Fresno, Redding, and Yountville homes, preventing it from gaining the full benefit of lessons learned: the ability during future similar projects to duplicate the successes and avoid the shortfalls experienced on earlier projects.

CalVet’s implementation plan stated that its PMO would lead lessons-learned sessions throughout the implementation of the system, to document any lessons that could be usefully applied to implementation at the next site. CalVet implemented the system in three phases: the pilot phase, which included headquarters and the homes in Barstow, West Los Angeles, Lancaster, and Ventura; the second phase, which included Chula Vista; and the final phase, which included Fresno, Redding, and Yountville. Documentation summarizing the lessons-learned sessions held shows that CalVet conducted 14 of these sessions, from its initial procurement efforts in March 2009 through implementation at Chula Vista in August 2013. According to CalVet’s summary of these sessions, they focused on what went well, what did not go as planned, and areas for improvement.

Given that CalVet implemented the system in phases, it could have used the lessons it learned during the pilot implementation to improve the Chula Vista implementation, and it could have used the lessons learned from both the pilot implementation and at Chula Vista to improve the final phase of its implementation. Specifically, CalVet’s summary from its lessons-learned session after the pilot implementation stated that staff indicated that there was a lack of on-site support during evening shifts during the implementation. Following the implementation at Chula Vista, CalVet’s lessons-learned session notes again identify that evening support was not consistent. Additionally, after the pilot implementation, CalVet’s lessons-learned summary identified concerns with training and noted as an improvement going forward the need to maintain binders with the training materials and processes in a central location. However, CalVet’s summary from its lessons-learned session for the implementation at the home in Chula Vista again notes the need to maintain materials in a central location. These examples demonstrate that CalVet repeated mistakes and therefore did not effectively use the lessons it learned. In fact, CalVet’s PMO manager identified only one change that CalVet did make as a result of lessons learned. Specifically, CalVet increased the time allotted for training at Chula Vista. She stated that CalVet did not go back and document how it implemented changes based on lessons learned, so she is unsure whether any other lessons learned were implemented.

CalVet did not conduct lessons-learned sessions following the final implementation phase at the Fresno, Redding, and Yountville homes, limiting its ability to identify problems that it could correct or avoid and successes it could repeat in future similar projects. According to the PMO manager, CalVet did not conduct those lessons-learned sessions because the system was not fully implemented at those three homes. However, CalVet’s implementation plan specified that the PMO would conduct lessons-learned sessions throughout the implementation, and since the homes still implemented modules of the system, we believe there were opportunities to document lessons learned for future projects. When we pointed out these missed opportunities to the PMO manager, she did not remember whether project management considered that lessons could be learned from its partial implementation at the three homes. However, she stated that it is reasonable to think so.

According to its PIER, CalVet captured final lessons learned during interviews with key management and executive staff. The report states that participants were given the opportunity to look back and identify the most significant problems and successes of the project that would benefit future CalVet project efforts. CalVet summarized these final lessons learned into eight categories, such as scope management, requirements or change management, contract management, and project governance, and it described findings and recommendations for each category. For example, one finding noted that CalVet did not actively manage the system contractor, resulting in the system contractor not always delivering services according to its contract, and agreed-upon deliverables were not reflected in contract amendments. The corresponding recommendation was to make sure the right resource is assigned to not only manage the contract but also to enforce the terms of the contract. In another example, CalVet’s finding stated that the requirements were loosely managed throughout the project life cycle, referring to its early identification of too many requirements, the removal of many requirements in the subsequent RFP, the significant gap between the two RFPs, and the large number of change requests. The related recommendation states that CalVet should establish a requirements management plan that defines the process of scheduling, coordinating, and documenting the requirements engineering activities including elicitation, analysis, specification, and verification. CalVet’s incorporation of these final lessons learned into its preparation for its planned future project is critical to ensure that the next system implementation is successful.

Recommendations

CalVet

To ensure that its project management of IT projects promptly identifies potential problems and develops resolutions, by September 2016 CalVet should define the project executive’s and project manager’s responsibilities to ensure that the individuals who fill those positions take an active role in each project.

To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016 CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management and other required plans. This formal process should also include a process to periodically verify that the project team is adhering to all these plans.

To ensure accountability and independence between the provision of IPO and IV&V services on future IT projects, by September 2016 CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by the Technology Department.

To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016 CalVet should establish a process to periodically verify that its staff follow state contracting requirements and maintain all required contract documentation.

To ensure it maintains all documentation related to its IT contracts, CalVet should, by September 2016, establish a process to verify that all divisions comply with its policy requiring each division to submit a records retention schedule to its Office of Procurement and Contracting.

To ensure that it only accepts deliverables and approves payment for deliverables that are complete and meet contract requirements, by September 2016 CalVet should establish processes to do the following:

• Ensure that the project executive verifies that individuals assigned to project roles are adequately qualified and experienced.
• Verify and maintain documentation of receipt of all contract deliverables before approving payment.
• Strengthen its contract management on all future projects by requiring the project manager to sign off on invoices along with the contract manager before approving payment.
  

To ensure that it maximizes its opportunity to successfully implement future IT projects, including its plan to replace its current system, CalVet should, by September 2016, establish a formal process to do the following:

• Document the changes it makes as a result of the lessons-learned sessions it conducts.
• Verify that its staff conducts lessons-learned sessions for all key phases of the next project.
• Incorporate the recommendations identified in its PIER.
  

Technology Department

To ensure that it can demonstrate that it is acting in the best interest of the State, the Technology Department should, by December 2016, create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies’ contract disputes and negotiations for the termination of a contract and maintain those documents according to its record retention schedule.

To ensure accountability and independence between the provision of IPO and IV&V services, the Technology Department should, by December 2016, establish a written policy requiring departments that request and receive approval to contract for IPO services to use a different contractor than the one providing IV&V services.
Although the Technology Department indicated that its intent is not to outsource its statutory responsibility for IPO, in any instances where its staff conduct a portfolio review of a project’s IPO, the Technology Department should, by December 2016, establish a process for its review of documents created by the agency’s IPO contractor that includes verifying whether these reports include critical analysis of project progress and vendor performance so it can intervene when necessary.

We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Respectfully submitted,

ELAINE M. HOWLE, CPA
State Auditor

Date: June 16, 2016

Staff:
Tammy Lozano, CPA, CGFM, Audit Principal
Richard D. Power, MBA, MPP
Fahad Ali, CFE
Brian D. Boone, CIA, CFE
Karen Jenks, MBA

IT Expert:
Catalysis Group

Legal Counsel:
J. Christopher Dawson, Senior Staff Counsel

For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255.

---

Footnotes

³ The PIER is a  report submitted to the Technology Department in support of an agency’s request to consider the project complete and to terminate project reporting. Go back to text.

⁴ Until July 2013, the California Department of Technology was known as the California Technology Agency, and before that it was the Office of the Chief Information Officer. Go back to text.

⁵ High Risk Update—California Department of Technology: Lack of Guidance, Potentially Conflicting Roles, and Staffing Issues Continue to Make Oversight of State Information Technology Projects High Risk, Report 2014‑602, March 2015. Go back to text.

Back to top