Report 2018-114 All Recommendation Responses

Report 2018-114: Disabled Veteran Business Enterprise Program: The Departments of General Services and Veterans Affairs Have Failed to Maximize Participation and to Accurately Measure Program Success (Release Date: February 2019)

Recommendation #1 To: General Services, Department of

To increase the number of DVBE firms that awarding departments can contract with when required to use LPAs, General Services should develop and implement a plan to encourage DVBE firms to participate in LPAs.

1-Year Agency Response

DGS Procurement Division (PD) has developed and fully implemented a plan to encourage DVBE firms to participate in LPAs.

1) DGS-PD created checklists for buyers to maximize opportunities for DVBE participation on Statewide Contracts. These checklists guide buyers through the process of identifying potential DVBE participants, both as prime contractors and as sub-contractors. Specific steps include targeted searches of the OSDS database and review of the identified DVBE's business and product information.

2) DGS-PD is awarding contracts by group or line item when appropriate to keep the size of contract awards to a level at which SBs (most DVBEs are also SBs) can comfortably participate. Examples include:

- Newsprint and Carbonless Paper Rolls

- Forklifts

- Traffic Control Modules

- Renewable Diesel

- Enterprise Technology

- Sand and Cinders

3) DGS-PD continues to educate DVBEs about tools available for their use within the California State Contracts Register (CSCR) called Vendor Ads to assist them to market their business or indicate a willingness to participate on a contract within the solicitation when advertised. LPA contracting opportunities are advertised on the CSCR. Information was presented about Vendor Ads and links to tutorials for their use at the Small Business Advisory Council on June 26, 2019, September 4, 2019, and December 4, 2019.

4) DGS-PD encourage DVBE participation in LPA bidding opportunities by informing them of upcoming solicitations and identifying those commodities where participation by DVBEs has historically been lower, where the bidder pool has shrunk, where group/line item awards exist or where there is ample subcontracting opportunity. This has been accomplished by posting upcoming bid opportunities on the DGS-PD website. DGS-PD presents this information quarterly at the Small Business Advisory Council. This information was presented on June 26, 2019, September 4, 2019, and December 4, 2019.

Completion Date: December 2019
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The Contracts Management Section (CMS) has created checklists outlining the steps to maximize opportunities for DVBE participation on Statewide Contracts. These checklists guide buyers through the process of identifying potential DVBE participants, both as prime contractors and as sub-contractors. Specific steps include targeted searches of the OSDS database and review of the identified DVBE's business and product information.

DGS/PD has continued to award contracts by group or line item when appropriate to keep the size of contract awards to a level at which SBs (most DVBEs are also SBs) can comfortably participate. Examples of such contract awards in the past six months include:

- Newsprint and Carbonless Paper Rolls

- Forklifts

- Traffic Control Modules

- Renewable Diesel

- Enterprise Technology

- Sand and Cinders

Solicitations are advertised on the CSCR through the Cal eProcure portal. One of the features of the CSCR is the posting of Vendor Ads soliciting or announcing availability as a sub-contractor. Post and View Vendor Ads is accessible from several different training modules. Training is also provided for user support, navigating the website, how to view solicitations, and utilizing Vendor Ads. Vendor Ads were discussed at the April 9, 2019, PART meeting, familiarizing state purchasing officials with the offered functionality. Additionally, the DGS/PD website features a page to familiarize vendors with the Cal eProcure portal, providing videos guiding users through the registration process and fully utilizing the CSCR.

DVBE inclusion in LPA Contracts was presented at the June 26, 2019, Small Business Advisory Committee meeting. The information provided included finding upcoming solicitations prior to advertisement in the CSCR, an overview of Vendor Ads, and introduced the new features of the State Contract Index Listing that detail remaining contract extensions and links to related solicitation postings.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

General Services' response does not address our recommendation. Specifically, General Services did not address whether it has developed and implemented a plan to encourage DVBE firms to participate in LPAs. Until it addresses the recommendation, we will continue to report the status as not fully implemented.

Auditee did not substantiate its claim of full implementation

60-Day Agency Response

The Department of General Services (DGS) continues to explore opportunities for Disabled Veteran Business Enterprise (DVBE) firms to participate in Leveraged Procurement Agreements (LPA) and identify services or goods that can be made available to DVBE firms in large scale contracts. Although LPAs are created to increase the buying power of the state and leverage economies of scale in purchasing goods and services, DGS is committed to encouraging DVBE firms to participate in LPAs.

DGS' plan to encourage DVBE participation on LPAs including where appropriate, breaking awards into groups or line items for smaller contract award amounts, which in turn will lead to the increased likelihood that Small Businesses (which includes most DVBEs) can successfully perform on the contracts. In industry conditions that are limited, DVBE participation incentives will be included to account for as much as five percent participation by DVBEs.

DGS created a tool in the Cal eProcure, California State Contracts Register (CSCR) solicitation advertisements, to include the ability for DVBE suppliers to post attachments to the solicitation ads announcing their availability to participate as primes or subcontractors.

Further, DGS' Procurement Division (PD) is increasing the regularity of sharing upcoming solicitations with DGS' Office of Small Business and DVBE Services (OSDS) and the DVBE Advisory Committee quarterly by attending all meetings. Upcoming solicitation opportunities will also be posted on PD's website.

DGS' Contract Management Units recently started to spend more time during the contract development phase identifying certified DVBEs that perform work within the scope of the procurement and will be sending out a notification when the solicitation is released.

Estimated Completion Date: December 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #2 To: General Services, Department of

To ensure that a greater number of DVBE firms benefit from the DVBE program, General Services should work with awarding departments to understand why only a few DVBE firms receive a large number of contract awards. Based on its findings, General Services should take the necessary steps to remedy this situation.

Annual Follow-Up Agency Response From October 2020

As of September 2020, the Department of General Services (DGS) has fully implemented this recommendation.

Through its collaboration with awarding departments and DVBE stake holders, DGS identified the top five reasons that have contributed to the problem.

To address the top five reasons, DGS' 2020 Targeted Outreach Plan has corrected the following:

- DVBEs do not list or provide the required products/services used by the state - As of September 2020, more than 31,492 UNSPSC codes have been removed, which will help DVBEs by limiting the codes set to only those that are useful to the state and increase the ability for the DVBE community to accurately identify what they are selling and to match that to what the state is looking to buy.

- DVBEs unknowingly list incorrect UNSPSCs/keywords - Outreach has launched webinar trainings which include information about selecting the appropriate keywords/UNSPSC codes the state uses.

- Desire to reuse qualified, proven DVBE firms - On April 28, 2020, DGS entered into an agreement with the California Department of Veterans Affairs, Veterans Services Division, for the primary purpose to help and promote self-reliance for California's DVBEs. CalVet also has hired two new DVBE advocates as part of the agreement to work closely with departments and DVBE firms.

- Difficulty getting DVBE responses - To remedy this, DGS conducted 160 outreach events with DVBEs and included information on how to update their contact information on their Cal eProcure profile and the importance of being responsive to buyers who contact them directly.

- Cal eProcure problems - DGS' Outreach and Certification Unit has identified and resolved more Cal eProcure defects and regularly tests the system to ensure it is working properly.

Completion Date: September 2020

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

Since launching of the Procurement Division's (PD) eProcurement Business Intelligence Strategies Section (eBISS) State Contract and Procurement Registration System (SCPRS) Dashboard website, OSDS has emphasized with state departments the importance of using the dashboards to identify areas where non-certified suppliers are being used and instead utilizing certified DVBE suppliers. This has simplified finding DVBE suppliers for departments and introducing buyers to DVBE firms that departments are not currently using. We will continue to emphasize these tools at quarterly advocate meetings, state agency visits and ongoing communications.

OSDS also added new awards for the annual State Agency Recognition Awards to commend state buyers who used the most variety of certified SB and DVBE firms during the last fiscal year. Presenting this award at a statewide event gives consistent messaging to buyers about the importance of using a variety of firms when doing procurement.

Finally, according to SCPRS data for 2019, we have improved in DVBE participation from the FY 2017-18 figures contained in the CA State Auditor's report.

- 80 more DVBE firms received contract awards from January 2019 through January 2020 than in FY 2017-18.

- $59 million more in contract spending was awarded to DVBEs than in FY 2017-18.

- $9.8 million more was awarded to DVBEs outside of the top 30 DVBE firms than in FY 2017-18.

- Although the top DVBE firm increased its percentage share of all state spending with DVBEs, spending otherwise became more equitably distributed, with the top 30 firms receiving 1.6% less in total spending than the remainder of DVBE firms that won contracts in FY 2017-18.

- When the top two firms are removed from both figures, all DVBEs outside of the top 30 firms received 3.3% more of DVBE contract awards than in FY 2017-18.

Completion Date: December 2019
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

General Services did not address our recommendation that it work with awarding departments to understand why only a few DVBE firms receive a large number of contract awards. Until it understands the reasons behind why only a few DVBE firms receive a large number of contract awards, it cannot develop effective remedies. Further, although General Services claims that it has improved DVBE participation in fiscal year 2018-19 compared to fiscal year 2017-18, it did not provide any documentation to support this claim.

Auditee did not address all aspects of the recommendation

6-Month Agency Response

On February 8, 2019, DGS issued Broadcast Bulletin P-04-19 informing state agencies and interested stakeholders of the launch of the Procurement Division's (PD) eProcurement Business Intelligence Strategies Section (eBISS) State Contract and Procurement Registration System (SCPRS) Dashboard website. The information from the dashboard will assist departments by allowing them to make more data-informed decisions regarding which DVBE firms to use and what potential goods and services are being purchased by other state departments from certified DVBE firms.

DGS continues to meet with state departments and DVBE advocates quarterly and by request to analyze why only a few DVBE firms receive a large number of contract awards and actively collaborates with them to increase the number of DVBE firms participating in state contracts. Since the April 23, 2019 SB/DVBE advocate meeting, DVBE participation is a standing agenda item and allows departments an opportunity to discuss strategies for increasing the number of firms receiving DVBE contract awards. Advocates are encouraged to report their department's needs to support this endeavor with the Outreach team on an ongoing basis.

In addition, all department advocates received a survey from Outreach in May 2019 asking them to identify potential reasons why only a few DVBE firms receive a large number of contract awards. The responses will provide the Outreach team with data to better assist state departments in contracting with a variety of qualified DVBE firms. As of July 8, 2019, 85 advocate survey responses have been collected, and Outreach uses the information collected to provide ongoing DVBE contracting guidance to departments. Outreach will continue to provide departments information on the eBISS SCPRS Dashboards and meet with departments to address DVBE contracting concerns.

Completion Date: June 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Partially Implemented

In response to the May 2019 survey that General Services conducted, state agencies cited various reasons to explain why only a small number of DVBE firms are receiving contract awards. The primary reason cited by state agencies was not having enough qualified DVBE firms in the state database that can meet the state agency's contracting needs. However, General Services' response does not address the steps it has taken to remedy the situation that only a few DVBE firms are receiving a large number of contract awards. Until it does so, we will continue to report this recommendation as partially implemented.

Auditee did not address all aspects of the recommendation

60-Day Agency Response

On February 8, 2019, DGS issued Broadcast Bulletin P-04-19 informing state agencies and interested stakeholders of the launch of the Procurement Division's (PD) eProcurement Business Intelligence Strategies Section (eBISS) State Contract and Procurement Registration System (SCPRS) Dashboard website. This new information from the dashboard will assist departments by allowing them to make more data informed decisions regarding which DVBE firms to use and what potential goods and services are being purchased by other state departments from certified DVBE firms.

DGS is continuing to meet with other state departments and DVBE advocates to analyze why only a few DVBE firms are receiving a large number of contract awards and will collaborate with them on opportunities to increase DVBE participation. Moving forward, all Small Business (SB) and DVBE advocate meetings will have DVBE participation as a standing agenda item (starting at the April 23, 2019 meeting), which will give departments an opportunity to discuss strategies for increasing the number of firms receiving DVBE contract awards. Advocates will be encouraged to report their department's needs to support this endeavor with the DGS Outreach Team on an ongoing basis.

The Outreach Team will use the information provided by advocates to create updated best practices and suggested strategies for DVBE contracting with the goal of a larger number of DVBE firms receiving contracts in the future. The advocate surveys will be collected by June 1, 2019 and the Outreach Team will update the best practices for DVBE contracting by January 1, 2020.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #3 To: General Services, Department of

To ensure that awarding departments can effectively identify DVBE firms that provide needed products and services, General Services should work with these departments to continue to narrow the codes available to those commonly used by awarding departments to more precisely identify what the State purchases and streamline search criteria in Cal eProcure.

1-Year Agency Response

As of January 2020, DGS PD has provided the shortened custom code set to the Financial Information System for California (FI$Cal). On January 14, 2020, PD has informed departments of this change at the Purchasing Authority Roundtable (PART) meeting. Training will be rolled out once FI$Cal has completed their portion of the solution.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

The DGS Procurement Division (PD) continues to work on refining an updated version of the United Nations Standard Product and Services (UNSPSC) set to be implemented in the Financial Information System for California (FI$Cal). Originally it was planned for a July 2019 release date; unfortunately, PD is pushing that deadline out to January 2020. This allows for PD to continue to provide Departments support and assistance with their backlog transactions, while not introducing anything new that could potentially slow that effort down.

The purpose of a refined code set is to customize it so that Departments are spending less time attempting to find codes for goods and services they purchase every day. The new code set will also allow Department buyers to be able to easily identify Suppliers, which provide those same goods and services, and increase data accuracy by limiting the number of choices to only those that are relevant to the State of California.

DGS PD will inform Departments of this change through the Purchasing Authority Roundtable (PART) meeting; SB/DVBE Advocate Quarterly meetings; Delegation Accreditation meetings, and this topic being part of the Cal Procurement Contract Academy (Cal PCA) Basic training module will be updated. This way new buyers, as well as the seasoned buyers and the supplier communities will all understand how to identify the correct UNSPSC to accurately reflect what the state buys.

Estimated Completion Date: January 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

The DGS Procurement Division is currently working with the updated version of the United Nations Standard Products and Services Code (UNSPSC) set to refine and build a custom code set that reflects what the state procures, which will be updated in the Financial Information System for California (FI$Cal) once it is finalized by July 2019. This refined code set will benefit state departments because they will spend less time searching for the correct codes, find vendors more easily, and increase the data accuracy by limiting the number of choices. DGS will inform departments through Purchasing Authority Roundtable (PART) meetings, SB/DVBE Advocate Quarterly meetings, Delegation Accreditation meetings, and is a topic of discussion in the Basic Acquisition training module as part of the Cal Procurement Contract Academy, the need to identify the UNSPSC codes they regularly use for purchases in order to accurately reflect what they buy.

Estimated Completion Date: July 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #4 To: General Services, Department of

To ensure that awarding departments can effectively identify DVBE firms that provide needed products and services, General Services should ensure that by October 2019 departments identify their contracting needs and then post these needs prominently on their websites as a resource for DVBE firms.

6-Month Agency Response

- Five departments complied with the request.

- Additional departments agreed to complete the report in the upcoming months.

As of June 2019, departments that had previously indicated to complete a Look-Ahead report before June 2019 were contacted by email.

- All departments that spent more than $10 million in FY 2017-18 and had not completed a Look-Ahead report - 35 departments in total - were contacted and invited again to create a Look-Ahead report.

- New efforts resulted in seven departments responding, including links to three new Look-Ahead reports.

- In total, 21 Look-Ahead reports have been developed and posted on departments' websites, and linked from the DGS "How to do Business with the State" web page at https://www.dgs.ca.gov/PD/Resources/Page-Content/Procurement-Division-Resources-List-Folder/How-to-do-business-with-the-state-of-California.

DGS will continue to encourage departments to share available Look-Ahead reports on their website as a resource for DVBE firms.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

In October 2018, DGS contacted state departments and encouraged them to post a Look-Ahead report of upcoming solicitations on their departments' website with a deadline of January 2019. A follow-up email was sent out in January, and a DGS Outreach Liaison attended the January Small Business (SB) and DVBE advocate meeting to explain the importance of providing upcoming solicitations to the DVBE firms. As of March 2019, five departments have complied to the request. Further, it was confirmed that additional departments have agreed to complete the report in the upcoming months. By October 2019, DGS will be able to identify the number of departments posting the Look-Ahead reports on their departments' website.

Estimated Completion Date: October 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #5 To: General Services, Department of

To ensure that awarding departments can effectively identify DVBE firms that provide needed products and services, General Services should continue to provide outreach services to DVBE firms on how to create effective keywords and choose appropriate codes.

6-Month Agency Response

Using the Statewide SB/DVBE Procurement Data Dashboard, Outreach randomly selected 50 DVBE firms that have won a state contract and analyzed their Certification profiles to collect data on what UNSPSC and keywords these firms included in their profile. Outreach then randomly selected 50 DVBE firms that had not yet been awarded a state contract and similarly analyzed the UNSPSC and keywords on their profiles.

In addition, Outreach sent all department advocates a survey in May 2019 asking for feedback on their department's best practices for utilizing UNSPSC and keywords when searching for DVBE firms. The survey also requested that departments give feedback on common mistakes DVBE firms make when completing their certification profiles, so that Outreach can better assist state departments in contracting with a variety of qualified DVBE firms. As of July 2019, 85 advocate survey responses have been collected by Outreach. Outreach also held discussions at the quarterly advocate meetings in April 2019 and June 2019 regarding UNSPSC and keyword best practices to further collect information from advocates.

Outreach utilized the independent research of DVBE certification profiles, survey results from the advocate surveys, and results of the discussions at the quarterly advocate meetings to update best practices for using UNSPSC and keywords. From those best practices, Outreach completed the following:

- New training slides on UNSPSC and keywords to be used in workshops.

- Updated web resource material and created a handout for DVBEs regarding UNSPSC and keyword best practices.

- Created a new section highlighting UNSPSC and keywords in the updated "How to Do Business with the State" brochure.

Outreach will continue to provide guidance to DVBE firms on UNSPSC and keyword best practices to better ensure awarding departments find certified DVBE firms who provide needed goods and services.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DGS' Office of Small Business and DVBE Services (OSDS) is currently providing information to DVBE firms about how to choose keywords and appropriate codes at Outreach Workshops and in the included materials that are disseminated at the outreach events and as web resources. Other state advocates are also doing the same.

By July 1, 2019, OSDS will do an analysis of a sample of currently certified DVBE certification profiles to identify common errors made in selecting keywords and UNSPSC codes. Using the information gathered, OSDS will incorporate an updated section of the Outreach Workshop presentation about selecting the most effective keywords and codes in order to ensure that awarding departments can effectively identify DVBE firms that provide needed products and services. The information from the presentation will also be made available as a web resource. The updated keyword and UNSPSC code training slides and web resource materials will be available by January 1, 2020.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #6 To: General Services, Department of

To ensure that awarding departments can effectively identify DVBE firms that provide needed products and services, General Services should explore other options for making the Cal eProcure search engine more effective for awarding departments.

1-Year Agency Response

As of January 2020, DGS PD has provided the shortened custom code set to the Financial Information System for California (FI$Cal). On January 14, 2020, PD has informed departments of this change at the Purchasing Authority Roundtable (PART) meeting that they can expect to see the new custom set roll out once FI$Cal has completed their portion of the solution.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

DGS is continuing its work with the Financial Information System for California (FI$Cal) to implement a succinct custom United Nations Standard Products and Services Code (UNSPSC) set and evaluate the effectiveness of the Cal eProcure search engine.

DGS implemented two dashboards that give departments the ability to view purchasing data within 24 hours of it being reported into SCPRS. The Statewide Small Business and DVBE Procurement Data Dashboards help the search engine by provided alternative ways for departments to get to information without having to use PeopleSoft functionality.

Estimated Completion Date: December 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Estimated Completion Date: December 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation for Legislative Action

To ensure that awarding departments that fail to meet the 3 percent goal receive the assistance necessary to achieve the goal, the Legislature should amend state law to transfer the responsibility for monitoring and assisting underachieving departments from CalVet to General Services.

Description of Legislative Action

As of April 2023, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: April 2023

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

As of February 14, 2022, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: February 2022

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

As of February 14, 2021, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: February 2021

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

As of February 2020, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: February 2020

California State Auditor's Assessment of 1-Year Status: No Action Taken

Description of Legislative Action

The Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: August 2019

California State Auditor's Assessment of 6-Month Status: No Action Taken

Description of Legislative Action

The Legislature has not taken any action to address this specific recommendation.

Legislative Action Current As-of: April 2019

California State Auditor's Assessment of 60-Day Status: No Action Taken

Recommendation for Legislative Action

To minimize the occurrence of program abuse involving DVBE subcontractors, the Legislature should amend state law to require awarding departments to notify those DVBE subcontractors when they are named on an awarded contract.

Description of Legislative Action

As of April 2023, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: April 2023

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

As of February 14, 2022, the Legislature has not taken action to address this specific recommendation.

Legislative Action Current As-of: February 2022

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

As of February 14, 2021, the Legislature has not taken action to address this specific recommendation.

AB 1365 (Chapter 689, Statutes of 2019) would have required an awarding department to directly inform a DVBE of its inclusion in an awarded contract when the DVBE has been identified as a subcontractor within the awarded contract. This provision was removed from the bill prior to its enactment.

Legislative Action Current As-of: February 2021

California State Auditor's Assessment of Annual Follow-Up Status: No Action Taken

Description of Legislative Action

Legislative Action Current As-of: February 2020

California State Auditor's Assessment of 1-Year Status: Legislation Proposed But Not Enacted

Description of Legislative Action

AB 230 (Brough), in part, would require a department awarding a disabled veteran business enterprise (DVBE) contract to include proof of payment for work done by the DVBE and the amount and percentage of work the prime contractor committed to provide to one or more DVBEs under the contract. The bill also states legislative intent to implement the recommendations in the audit.

AB 1365 (Assembly Veterans Affairs Committee), in part, would require an awarding department to directly inform a DVBE of its inclusion in an awarded contract when the DVBE has been identified as a subcontractor within the awarded contract.

Legislative Action Current As-of: August 2019

California State Auditor's Assessment of 6-Month Status: Legislation Introduced

Description of Legislative Action

Legislative Action Current As-of: April 2019

California State Auditor's Assessment of 60-Day Status: Legislation Introduced

Recommendation #9 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should assess, at least annually, the effectiveness of its past outreach efforts in increasing the number of DVBE firms that become certified.

Annual Follow-Up Agency Response From October 2021

Through the establishment of two staff through an inter-agency agreement with DGS, and a restructured DVBE Advisory Council, CalVet conducted a review of its DVBE outreach. Based on the assessment it was determined that there are several areas of opportunity. The analysis revealed two primary findings:

- There was not an adequate means of tracking inquiries from current and prospective DVBEs made to CalVet.

- There was not an adequate means of determining direct feedback from awarding departments regarding the necessary goods and services they are seeking.

With this information CalVet created the Request for Response survey hosted on its website and a direct email that CalVet staff monitors on a daily basis. The Request for Response survey and the DVBE email are avenues for DVBE community to gain increase access to state advocates, as well as communicate with CalVet in an effort to improve communication. This allows CalVet the ability to refer current and prospective DVBEs directly to the Department of General Services (DGS) certification officers to begin the process of acquiring their certification, and maintain communication and follow up with the veteran throughout the process.

Additionally, CalVet developed the following surveys in order to gain further resolution on the needs of awarding departments to further increase outreach efficacy:

1. A certified DVBE survey that identifies goods and services that certified DVBEs offer.

2. A survey that identifies what goods and services awarding departments are struggling to find DVBE vendors

CalVet will conduct an annual assessment for DVBE outreach as well as use the information gathered from these surveys to connect awarding departments directly to the DVBEs that offer the goods and services they are seeking.

Completion Date: October 2021

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

On April 1 2020, VSD established an Interagency Agreement (IA) with DGS to provide CalVet the means to implement the recommendations of this audit. The IA includes two full time analyst positions which were fully staffed on October 1, 2020. The essential functions of these positions will assist the department in analyzing outreach efforts by:

-Working directly with DVBE Advocates throughout the state to discover the procurement needs of awarding departments. CalVet has begun the process of training these staff working collaboratively with DGS. This will ensure CalVet's recruitment of DVBEs is strategic and works within business sectors that give DVBEs the greatest likelihood of success.

-Placing the two DVBE positions with the California Transition Program will provide greater support to these two positions through resources such as survey monkey, research staff, and outreach.

-The establishment of the IA includes language for data sharing between the two departments with the first request for new DVBE certification data anticipated to commence on November 1, 2020. This will allow for the identification of a reliable benchmark to measure the success of DVBE-specific outreach through regular periodic surveys.

-The DVBE Advisory Council has been revamped significantly to ensure programmatic issues brought forth to the council are relevant and cycle through an efficient process which produces actionable recommendations for the department to address.

It is through the DVBE Advisory Council that the efforts of these organizations and individual DVBEs be coordinated, which will ensure growth of the program. The DVBE Advisory Council will serve as a culturally competent entity that will perform analysis of DVBE outreach activities to increase program participation.

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

CalVet underwent leadership changes that directly impacted the DVBE program in late 2019. These changes also include a deviation from the previously submitted DVBE Promotion and Outreach Enhancement Plan. In January 2020, the DVBE program responsibilities were transferred to the Veterans Services Division (VSD). VSD is establishing a new Interagency Agreement (IA) with DGS effective April 01, 2020 to provide CalVet the means to implement the recommendations of this audit. The IA includes 2 full time analyst positions whose essential functions will assist the department in analyzing outreach efforts by:

- Working directly with DVBE Advocates throughout the state to discover the procurement needs of awarding departments. This will ensure CalVet's recruitment of DVBEs is strategic and works within business sectors that give fledgling DVBEs the greatest likelihood of success.

- Performing analysis of certification and procurement data provided by DGS. This will allow for the identification of a reliable benchmark to measure the success of DVBE-specific outreach through regular periodic surveys.

- Coordinating a revamped DVBE Advisory Council to ensure that outreach strategies developed are culturally competent, relevant, and implemented through a community-based infrastructure.

DGS's annual analysis of their past outreach efforts recommends that they "continue to strengthen relationships with DVBE-focused organizations." It is through the DVBE Advisory Council that the efforts of these organizations and individual DVBEs be coordinated, which will ensure growth of the program. The DVBE Advisory Council will serve as a culturally competent entity that will perform analysis of DVBE outreach activities to increase program participation.

CalVet will also ensure our newly established internal audits unit will be an integral part in ensuring full and timely implementation of the recommendations provided moving forward.

Estimated Completion Date: 10/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

CalVet held further internal Discovery Meetings with DGS regarding previous DVBE promotional and outreach efforts. We identified the need for a more robust DVBE marketing plan. CalVet then developed a 2019 DVBE Promotion and Outreach Enhancement Plan (the Plan) that includes annual assessments as recommended (Attachment A). CalVet is in the initial phases of implementing the Plan. CalVet will continue to conduct Discovery Meetings with DGS and underperforming departments.

Estimated Completion Date: March 27, 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. In July 2018, CalVet reassigned the DVBE State Advocate to the Division of the Deputy Secretary of Minority and Underrepresented Veterans and began assessing past outreach efforts with internal and external "Discovery Meetings." The intent of the discovery meetings is to provide effective communication amongst stakeholders from CalVet to DGS to the implementing agencies and departments.

More specifically, these meetings evaluate past outreach efforts in order to understand systematic challenges within awarding departments. The results from the internal and external Discovery

Meetings include:

1. Modifying and formatting monthly DVBE reports to enable CalVet to track valid and invalid DVBE certifications, inspect inconsistencies, and target key DVBE industries for awarding agency needs;

2. CalVet will develop outreach events based on one-on-one awarding agency Discovery Meetings. CalVet noted targeted matchmaking for both awarding agencies and DVBEs;

3. CalVet will be identifying approaches to non-traditional methods of veteran businesses outreach. Target areas include: architectural, engineering, general contracting, facility management, building, management, and supplier management organizations;

4. Currently, CalVet is assessing new tools to measure DVBE success with targeted online surveys and batch surveys to be performed after every outreach event and bi-annually. The surveys will be distributed in June 2019.

5. The CalVet DVBE Advisory Council was reconstituted with a reorganized set of advisory board members, to better consider and facilitate CalVet's efforts with: effective DVBE outreach, DVBE mentorship with seasoned prime DVBEs as mentors, and to facilitate efforts with aligning qualified DVBEs with agencies that need to increase their DVBE goals. The first meeting occurred in February and additional meetings have been scheduled for calendar 2019.

Estimated Completion Date: July 31, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #10 To: General Services, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, General Services should assess, at least annually, the effectiveness of its past outreach efforts in increasing the number of DVBE firms that become certified.

1-Year Agency Response

Since February 2019, OSDS has been collecting data on DVBE-focused Outreach activities. Based on the data collected, Outreach has completed an annual assessment as of January 2020 on the effectiveness of past outreach efforts in increasing the number of DVBE firms that become certified that can provide the necessary goods and services awarding departments are seeking. This completed assessment revealed which activities were impacting the program most positively and which needed adjustments moving forward. This can be seen in Outreach's targeted Outreach plan, completed in January 2020, which commits to focus on the following going forward:

- Outreach will continue to attend events hosted by our DVBE strategic partners and provide direct support to DVBEs and potential DVBEs.

- Continue to strengthen relationships with DVBE-focused organizations and request DVBE organizations to focus outreach to DVBE firms that can provide goods and services state departments are seeking.

- Outreach is launching a media campaign to capture DVBE firm success stories in contracting.

- Address difficulties using Cal eProcure's search function and certification application.

- OSDS will continue to host quarterly meetings for SB/DVBE advocates, invite advocate feedback to improve outreach, host annual agency visits with departments who struggle to find DVBE firms who provide the goods and services they need, and continue to provide training.

- Outreach will continue to revamp brochures and online resources to provide the necessary information in the clearest and most concise manner.

- Monitor SCPRS data and dashboards to ensure that state departments are continuing to use a variety of DVBEs.

- Monitor DVBE certification data to see what types of goods and services newly certified DVBE firms provide to ensure they align with states needs and continue to evaluate if targeted outreach efforts are effective.

- Designing training webinars for State certified businesses.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

After Outreach conducts a certification workshop, all attendees are invited to fill out a post-event survey in order to assess the effectiveness of the workshop and how outreach can improve. These responses are evaluated and inform the content of future trainings.

As of January 2019, after all DVBEs complete their certification application online, they are given a survey asking how they found out about the certification program, and if applicable, what outreach event they attended. As of July 9, 2019, 1,750 responses have been collected, with nearly 23 percent of respondents identifying some outreach activity as their information source for obtaining certification.

Outreach developed a questionnaire that was distributed to six DVBE-focused strategic partners. This survey seeks information from DVBE-focused organizations on how Outreach can better address challenges to the disabled veteran business community in obtaining certification, bidding, and winning contracts and how DGS can improve the effectiveness of outreach efforts moving forward. As of July 8, 2019, responses have been received from three of these organizations. Outreach has followed up with organizations that have not responded and will continue to do so.

By January 1, 2020, the Outreach team will analyze all available gathered data regarding the effectiveness of DVBE outreach in 2019. Outreach will use the assessment to create an outreach plan for 2020 that will better address the needs of contracting state departments and the DVBE community.

Estimated Completion Date: January 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

To assist DGS with its assessment of goals of DVBE firms in becoming certified or to identify reasons they may choose not to become certified, surveys were sent to strategic partners to help assess the program and receive input on how to better serve the DVBE community. DGS' Outreach Team is collecting and tracking businesses owned by disabled veterans that are attending the outreach events. In addition, certification data is being collected and will be reviewed to determine whether these businesses obtained their certifications. As of January 2019, after DVBEs complete their certification application online, they are given a survey asking how they found out about the certification program, and if applicable, what outreach event they attended. Further, the outreach team will be collaborating with the California Department of Veterans Affairs (CalVet) to develop a survey that will be sent to DVBEs who received their certification last year.

By January 1, 2020, the Outreach Team will analyze all available gathered data regarding DVBE participation at outreach and certification events. The data will be assessed for the effectiveness of the 2019 outreach efforts and for increasing the number of DVBE firms that become certified. OSDS will use the assessment to create an Outreach Plan for 2020 that will better address the needs of state departments and the DVBE community.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #11 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should work with awarding departments to identify the types of goods and services for which they struggle to find a DVBE contractor or subcontractor.

Annual Follow-Up Agency Response From October 2022

In order to ensure the efficacy of CalVet's outreach efforts with awarding state departments, and to gain a better understanding of awarding department's challenges in awarding DVBE contractors and subcontractors, CalVet has developed and implemented a number of tools and best practices to support and maintain this ongoing effort. For continued knowledge transfer from awarding departments, CalVet developed the Annual State DVBE Advocate Survey (Advocate Survey). The annual Advocate Survey allows awarding agencies to provide the top 5 goods, and top 5 services that they struggle to procure from DVBEs. Information captured from this survey has been disseminated to the DVBE community, and will continue to be shared annually, in order to enhance the DVBEs awareness of what goods and services awarding departments desire to procure. Survey data is additionally used by CalVet in order to improve its outreach practices. The inter-agency agreement between CalVet and DGS continues to allow a dedicated CalVet position to collaboratively work alongside DGS in a supporting role as an SB/DVBE Advocate both within the structure of the California Small Business/Disabled Veteran Business Enterprise Advocate Steering Council and as a continuous resource for DVBE advocates. As such, CalVet analysts developed a process for the review and assessment of improvement plans with underperforming agencies, visitations and recommendations directly with underperforming departments, as well as the development of procedures for DVBE exemption requests provided by DGS. CalVet continues to utilize and refine tools such as the Request for Response that offer state advocates assistance in locating DVBEs that provide goods and services that departments struggle to find.

Completion Date: October 2022

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2021

CalVet has recognized the need to implement a yearly survey for awarding departments and the challenges they experience awarding DVBE contractors and subcontractors that provide the goods and services they seek to procure in order to gain a base line understanding of their needs. In response, CalVet has developed an annual Advocate Survey that will allow awarding agencies to provide the top 5 goods and top 5 services that they have challenges procuring from DVBEs, as well as to provide CalVet any additional information salient to their department's procurement needs. Information captured from this survey, will be shared with the DVBE community of what goods and services are opportunistic for their business as well as by CalVet in order to improve its outreach practices. The inter-agency agreement also allows there to be a position dedicated to working alongside DGS and awarding department Advocates on Goods & Services they seek and supporting their role as an SB/DVBE Advocate both within the structure of the California Small Business/Disabled Veteran Business Enterprise Advocate Steering Council and as a year round resource for DVBE advocates. In order to fully execute this recommendation, CalVet analysts are in the process of implementing procedures for the review and assessment of improvement plans in coordination with DGS for underperforming agencies, visitations with underperforming departments, as well as developing procedures for DVBE exemption requests that are sent from DGS. The analyst have also continue to develop tools such as the Request for Response in order to provide advocates a direct line to CalVet in order to gain assistance in locating DVBEs that provide goods and services that departments struggle to find.

Estimated Completion Date: October 1, 2022

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2020

Due to complications from COVID 19, the two positions funded through the IA with DGS were not filled until October 1, 2020. Since then, the two funded positions have begun the process of working directly with DGS in establishing CalVet's role in supporting State DVBE Advocates and underachieving state departments. These analysts are working to establish year-round engagement with departments who have struggled to reach their contracting goals by providing direct support to connect with potential DVBE contractors offering the goods and services they seek. Further, the data sharing agreement between CalVet and DGS established through the IA will further increase CalVet's ability to further assess its activities to ascertain program impact.

Documented Evidence:

Duty Statements (signed) for the two DVBE employees

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

The new IA will greatly increase connectivity to the statewide Advocate Program outside of the quarterly meetings held in coordination with DGS. The new positions will provide continuous support to the DVBE advocates in underachieving departments by acting as a conduit to connect DVBEs with the Look Ahead Reports developed through the efforts of DVBE advocates working with DGS. These analysts will ensure year-round engagement with departments who have struggled to reach their contracting goals by providing direct support to connect with potential DVBE contractors offering the goods and services they seek. This will greatly increase CalVet's level of engagement with underperforming departments and with an increase in data sharing between the two departments, CalVet will have the ability to further assess its activities to ascertain program impact and success.

Estimated Completion Date: 12/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

During the Discovery Meetings we worked with multiple awarding departments to identify the types of goods and services for which they struggled to find a DVBE contractor or subcontractor. One issue we identified was that the departments did not have a good process for tracking such goods and services. An identified best practice is for departments to develop a 12-Month Look Ahead Report. These reports are intended to project the goods and services a department will likely utilize in the coming year.

CalVet intended to review the reports with the awarding department to identify the goods and services underserved by DVBEs. This exercise revealed the need for improved operational effectiveness between DVBE advocates and their purchasing teams. Therefore, CalVet is in the early stages of developing a statewide DVBE training for department advocates and their purchasing team.

Estimated Completion Date: March 27, 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. During January 2019, CalVet scheduled a series of aforementioned Discovery Meetings with agencies not meeting the 3 percent requirements, as well as with entities that have expressed a desire to increase their DVBE participation goals beyond the 3 percent requirement.

During each of the individual agency Discovery Meetings, a "DVBE Roadmap and Success Timeline" and a simplified certified DVBE spreadsheet was given to each agency. This simplified search method should enable potential awarding agencies to locate certified DVBEs and potential DVBEs to fulfill their product and service needs.

Note:

- The refined list of DVBEs that CalVet generates is specifically tailored to meet each of the awarding agency's purchasing needs;

- To align state agencies with DVBEs, CalVet conducts monthly calls to DVBEs to support awarding agencies' contracting needs when an agency is experiencing difficulty trying to locate a specific DVBE for a particular contract; and

- Additionally, CalVet will work with DVBEs by email and phone to ensure their capabilities and the region-service information served by the DVBE are up-to-date. CalVet intends to share information regarding specific contracting opportunities and give guidance where appropriate to the DVBE to help them assess whether they are qualified to bid on the contract.

To assist the agencies and departments, CalVet has also developed a list of Veteran Service Organizations, categorized regionally. This provides DVBE awarding departments access to local veteran-centric organizations that might have a network of veteran-owned firms which are not currently certified as DVBEs. The goal is to connect this larger universe of veteran business owners with knowledge about the process and benefits of becoming DVBE certified as part of CalVet's efforts to increase the number of California DVBEs.

Estimated Completion Date: August 30, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #12 To: General Services, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, General Services should work with awarding departments to identify the types of goods and services for which they struggle to find a DVBE contractor or subcontractor.

6-Month Agency Response

DGS continues to meet with state departments and advocates to discuss goods and services they struggle to procure from DVBE contractors or subcontractors. Since April 2019, each SB/DVBE advocate meeting has included DVBE participation as a standing agenda item and has allowed departments an opportunity to discuss their specific department needs for DVBE goods and services and share strategies with other departments for finding DVBE firms. Advocates are encouraged to report their department's needs to support this endeavor with the Outreach team on an ongoing basis.

In addition, all department advocates received a survey from OSDS in May 2019 asking them to identify what struggles and best practices their department's buyers had in procuring from DVBEs, so that the Outreach team can better target DVBE firms that align with departments' needs. As of July 8, 2019, 85 advocate survey responses have been collected. Outreach is using the information to provide ongoing DVBE contracting guidance to departments.

DGS will also continue to address specific department needs for DVBE contractors and subcontractors during agency visits with departments who fail to meet their SB and/or DVBE goals. Departments are encouraged to contact the Outreach team throughout the year for DVBE contracting assistance.

Completion Date: June 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DGS' Outreach Team will continue to maintain communication with all state departments to help identify what types of goods and services departments struggle with finding DVBE firms to provide.

Moving forward, all SB/DVBE advocate meetings will have DVBE participation as a standing agenda item (starting at the April 23, 2019 meeting), which will give departments an opportunity to discuss their specific department needs for DVBE goods and services. After the April 23, 2019 meeting, all department advocates will receive a survey from OSDS asking them to identify their department's top DVBE areas of need, so that the Outreach Team can adjust and target DVBE firms that more closely align with departments' needs. The survey data provided by advocates will be collected and analyzed by July 1, 2019. This data will be used to create a plan for targeted DVBE outreach in order to better serve the needs of state departments. This plan will be finalized and implemented by January 1, 2020.

OSDS will also continue to address specific department needs for DVBE contractors and subcontractors during all agency visits with departments who fail to meet their SB and/or DVBE goals. All departments are encouraged to contact the DGS' Outreach Team throughout the year if they need any SB or DVBE contracting assistance.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #13 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should develop an outreach plan to include outreach activities found to be effective in the past based on its assessment. This plan should also emphasize outreach to increase the number of DVBE firms that provide the types of goods and services that awarding departments struggle to obtain from DVBE firms.

Annual Follow-Up Agency Response From October 2022

CalVet developed several mechanisms to gain a deeper understanding of the needs of DVBEs and prospective DVBEs, as well as the needs of state agencies and departments. To build a comprehensive, data driven outreach plan, CalVet developed several surveys directed toward multiple respondents to capture feedback and inquiries made to CalVet regarding the DVBE program. CalVet conducts a monthly survey of newly certified DVBEs to ensure that firms are aware of the resources available to them since becoming certified, state contracting opportunities, best practices when procuring with the state, and user awareness of Cal eProcure and its vital role in the procurement process connecting their business with state purchasers. The DVBE State Advocate Survey was also designed and implemented to identify areas where CalVet can outreach and assist agencies and departments in procuring from DVBEs. In addition, to enhancing outreach and training for existing and new DVBEs. Survey question content was specific to capturing goods and services awarding departments struggle to procure from DVBEs. CalVet also created the Request for Response survey which allows the DVBE community, small businesses, state advocates, contractors, veterans, and the general public to contact CalVet regarding the State of California's DVBE program. This survey provides key insights on the specific needs of the respondents, and allows CalVet to continually evolve it's outreach efforts to target those specific needs. In addition to surveys, CalVet established monthly meetings with certification officers from DGS-OSDS to effectively track and engage with those who expressed specific interest in becoming a certified DVBE, resulting in a 25% success rate of those veteran businesses becoming certified DVBEs. Over the last 12 months, CalVet has utilized data and information collected through the described mechanisms to develop a data driven outreach plan to further support of the DVBE program.

Completion Date: October 2022

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2021

In order to support the development of a comprehensive outreach plan, CalVet has implemented several instruments to gain a deeper understanding of the needs of established and prospective DVBEs, as well as state agencies. Assessments conducted by CalVet over the last year found that there were several opportunities to improve outreach efforts. In response, CalVet implemented several surveys to capture feedback and inquiries made to CalVet about the DVBE program. The purpose of the Certified DVBE Survey is to gather information from newly certified Disabled Veteran Business Enterprise firms. The purpose of the Advocate Survey is to identify areas where CalVet can assist departments in procuring from DVBEs. In addition to enhancing outreach and training for existing and new DVBEs. Survey question content was specific to capturing goods and services awarding departments struggle to procure from DVBEs. Another tool developed is the Request for Response survey which allows the DVBE Community, small businesses, state advocates, contractors, veterans, and the general public to contact CalVet regarding the State of California's DVBE program. CalVet also established a collaborative effort with certification officers from DGS-OSDS to effectively track and engage with those who expressed specific interest in becoming a certified DVBE. Over the next 6 months, CalVet will use the data and information collected through these instruments to develop a data driven outreach plan to further support of the DVBE program.

Estimated Completion Date: July 1, 2022

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2020

CalVet is currently reviewing the recently released DGS's Consolidated Annual Report (CAR) for FY 18-19. With the completion of hiring of DVBE staff on October 1, 2020, work will continue to prioritize which underachieving departments could benefit the most from CalVet support as well as establishing a method to support all DVBE advocates in their respective departments. The staff funded through the IA are currently working with DGS and department DVBE Advocates through DGS's advocates program and will perform research on the goods and services all awarding departments struggle to obtain from DVBE firms.

The areas of need identified will also allow CalVet to work with the revamped DVBE Advisory Council to establish a meaningful, community-informed outreach strategy that works to grow the number of DVBEs across the state. This strategy will also include recommendations on how to disseminate real-time procurement opportunities to DVBEs, as well as mitigate challenges by acting as a culturally competent bridge between DVBEs and state contract opportunities.

Documented Evidence:

1) Duty Statements (signed) for the two DVBE employees

2) CalVet DVBE Advisory Council Process Draft

3) DVBE AC Process Flow Draft

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

In 2019, the Statewide DVBE Advocate, in coordination with DGS, hosted nine "Discovery Meetings" with underachieving departments as identified within the DGS Consolidated Annual Report FY 2017-18. These meetings informed DVBE advocates about best practices for finding qualified DVBEs and encouraged underachieving departments to participate in multiple community-based DVBE outreach events. Particular focus was concentrated on identifying those goods and services that awarding departments struggle to find through a DVBE contractor or subcontractor. Currently, CalVet anticipates the release of DGS's Consolidated Annual Report (CAR) for FY 18-19, which will help to evaluate the impact of these past outreach efforts. CalVet will continue visiting underachieving departments through 2020, upon implementation of the IA, to prioritize which underachieving departments could benefit the most from CalVet support.

The staff funded through the IA will perform research on the goods and services all awarding departments struggle to obtain from DVBE firms and compare this information to the procurement and certification data that will be made available from DGS and advocates. This will allow CalVet to establish baseline of areas of need. CalVet will then actively work to connect state DVBE advocates with DVBEs that can assist awarding departments in meeting their 3% goal.

The areas of need identified will also allow CalVet to work with the revamped DVBE Advisory Council to coordinate the outreach and certification efforts of DGS and DVBE-focused organizations to establish a meaningful, community-informed outreach strategy that works to grow the number of DVBEs across the state. This strategy will also include the dissemination of real-time procurement opportunities, as well as mitigate challenges by acting as a culturally competent bridge between DVBEs and state contract opportunities.

Estimated Completion Date: 12/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

CalVet continues to implement DVBE Promotion and Outreach Enhancement Plan. For example, CalVet designed a tailored outreach event for CalTrans District 6 in Fresno, CA on May 22, 2019.

Approximately 50 vendors and 150 DVBEs and small businesses attended. CalVet facilitated one-on-one matchmaking sessions between DVBEs and state and local government buyers. On July 23, 2019, CalVet sent surveys to participating DVBEs. We asked them to evaluate the effectiveness of this event. The responses to the survey were submittted and are now under review within CalVet and DGS.

Completion Date: May 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

CalVet provided a draft outreach plan that includes a schedule for outreach events during fiscal year 2019-20. However, it has not yet assessed the effectiveness of its past outreach efforts in increasing the number of DVBE firms that become certified. Without such an assessment, it cannot ensure that its outreach efforts will be effective and will result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking.

Auditee did not substantiate its claim of full implementation
Auditee did not address all aspects of the recommendation

60-Day Agency Response

CalVet agrees with this recommendation. Since January 2019, CalVet has scheduled five meetings with awarding agencies. CalVet will continue to develop a "DVBE Roadmap and Success Timeline" based on assessments of each agency. Tailored outreach events are being designed to suit the awarding department's needs accordingly. An example is below:

CalVet is currently running a DVBE Collaborative pilot program in Fresno County. The DVBE Collaborative consists of sister agencies from federal, local and state governments, and local non-profits that have a special interest in ensuring that veteran businesses are successful. These agencies include the following: Small Business Administration, Small Business Outreach Centers, Veteran Business Office Centers, Small Business Development Centers, Procurement Technical Assistant Centers, SCORE, and local non-profits and small business incubators. The DVBE Collaborative pilot program was created to bring awareness to the DVBE Program, increase the number of DVBEs, and align government agencies with qualified DVBEs based on their industry.

This first DVBE Collaborative pilot program meeting was held in February 2019. Two additional meetings took place in March and April. The meetings focus on strategic DVBE outreach planning as well as work to facilitate a targeted DVBE outreach and matchmaking event on May 22 at a local Fresno small business incubator. CalVet will review outcomes of that pilot program at the end of the year.

CalVet is developing pre-event and post-event surveys. Each survey will evaluate the effectiveness of contracting opportunities, allow CalVet to gauge which DVBE firms will be in attendance, follow-up with newly certified DVBEs that attended the event, and gauge contracting opportunities that DVBEs were able to accomplish with prime contractors.

Estimated Completion Date: December 27, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #14 To: General Services, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, General Services should develop an outreach plan to include outreach activities found to be effective in the past based on its assessment. This plan should also emphasize outreach to increase the number of DVBE firms that provide the types of goods and services that awarding departments struggle to obtain from DVBE firms.

1-Year Agency Response

- Outreach will continue to attend events hosted by our DVBE strategic partners and provide direct support to DVBEs and potential DVBEs.

- Outreach is launching a media campaign to capture DVBE firm success stories in contracting.

- Address difficulties using Cal eProcure's search function and certification application.

- Outreach will continue to revamp brochures and online resources to provide the necessary information in the clearest and most concise manner.

- Monitor SCPRS data and dashboards to ensure that state departments are continuing to use a variety of DVBEs.

- Designing training webinars for State certified businesses.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

To assess outreach, the Outreach team distributes a Post-Workshop Evaluation Questionnaire to all attendees. This questionnaire helps determine if the businesses attending outreach workshops are certified or plan to get certified, as well as what additional outreach services would better support them with state contracting efforts. Of the workshop attendees who have taken the survey, 78% of businesses reported they are not certified and 93% of those plan to get certified.

Furthermore, Outreach composed an article that was disseminated to our strategic partners along with a request to include it in their membership newsletters and websites. As of June 2019, the article has been featured in ten strategic partner newsletters. The DGS article features DGS certification program information and the Outreach team's mission and goals in serving the SB/DVBE communities. DGS will continue to track DVBE outreach information posted and utilize our partners' ability to reach DVBE firms that provide the goods and services state departments need.

Lastly, Outreach has updated the Strategic Partnership Request form for FY 19-20 to reflect increased requirements for DVBE organizations we partner with going forward. These updated requirements ask DVBE organizations to:

- Provide post-outreach event information about potential DVBE firms who were in attendance.

- Focus outreach to DVBE firms that can provide goods and services state departments are seeking.

- Disseminate DVBE certification information to their membership.

- Regularly share state contracting needs with their membership.

DGS will continue to work closely with our strategic partners to identify strategies for finding additional DVBE firms that provide the goods and services state departments are seeking.

Estimated Completion Date: January 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

DGS will work closely with DVBE strategic partners such as the Small Business Development Centers (SBDC), Procurement Technical Assistance Centers (PTAC), etc. and reciprocity partners (various California cities, counties and public agencies) to identify strategies for finding additional DVBE firms that meet the needs of state departments. The Outreach Team will incorporate these strategies into developing a targeted outreach plan by January 1, 2020. This plan will include ways to leverage DVBE strategic partnerships to further support outreach efforts and wider dissemination of DVBE information to their member/suppliers.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #15 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should develop better tracking of the businesses owned by disabled veterans that attend its outreach events and review certification data to determine whether these businesses obtained their certifications.

Annual Follow-Up Agency Response From October 2021

Over the last year CalVet has put into place procedures for an ongoing analysis of those veterans who attended DVBE outreach events to identify those that have attended and since become DVBEs. This process ensures proactive support and follow up to the current and prospective DVBEs that come in contact with the department. CalVet has also established a collaborative effort with certification officers from DGS to effectively track and engage with those who expressed specific interest in becoming a certified DVBE. Through this partnership CalVet has developed a tracking system along with a follow-up protocol to further assist those veterans interested in becoming DVBEs through the use of greater data sharing with DGS.

Completion Date: June 2021

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

Through the establishment of the new IA CalVet now has formalized access to certification and procurement data. Due to impacts related to COVID-19, CalVet will begin receiving newly certified DVBE data from DGS on a periodic basis starting no earlier than November 01, 2020. This information will be used to evaluate the impact of our outreach through other veteran benefit related programs at the department to determine if these efforts led to any veteran businesses obtaining a DVBE certification. This includes programs such as the California Transition Assistance Program and the Local Interagency Network Coordinator program. Both of these programs interface with service members, veterans, and their families throughout the state. These programs work to connect veterans with their earned benefits across the state by providing direct referrals to services either in-person, over the phone, via email, for requests such as disability claims assistance, housing, healthcare, employment, education, financial assistance, and requests for military records.

Documented Evidence:

1) Interagency Agreement (executed)

2) CalTap Information (website link)

3) Local Interagency Network Coordinator Program Information (website link to CalVet Resource Book)

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

CalVet is establishing a new IA to formalize access to certification and procurement data. Beginning April 01, 2020 CalVet will begin receiving newly certified DVBE data from DGS on a periodic basis. This information will be used to evaluate the impact of our outreach through other veteran benefit related programs at the department to determine if these efforts led to any veteran businesses obtaining a DVBE certification. This includes programs such as the California Transition Assistance Program and the Local Interagency Network Coordinator program. Both of these programs interface with service members, veterans, and their families throughout the state. These programs work to connect veterans with their earned benefits across the state by providing direct referrals to services either in-person, over the phone, via email, for requests such as disability claims assistance, housing, healthcare, employment, education, financial assistance, and requests for military records.

Estimated Completion Date: 12/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

Our DVBE Outreach and Promotion Enhancement Plan 2019 includes the process of collecting attendee registration information at all DVBE events CalVet attends. CalVet uses Constant Contact to track registration for events developed by our department. We also use post-event surveys designed to determine whether the businesses obtained their certifications.

Estimated Completion Date: March 27, 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. CalVet developed a DVBE Enhancement Plan to track DVBE certification growth. The DVBE Enhancement Plan also includes an updated CalVet pre/post survey on outreach efforts and effectiveness. The intention is that DVBEs attending agency and department DVBE outreach events will be given the survey to inquire on the effectiveness of the outreach by the awarding state departments to DVBE firms. The survey will track the DVBE certification of veteran-owned business and prime/subcontractor opportunities. CalVet will follow up with attendees to see if they have completed their surveys.

Estimated Completion Date: October 31, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #16 To: General Services, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, General Services should develop better tracking of the businesses owned by disabled veterans that attend its outreach events and review certification data to determine whether these businesses obtained their certifications.

6-Month Agency Response

Since January 2019, Outreach has collected a list of disabled veteran owned businesses at each outreach event. As of June 2019, Outreach has collected data from 159 businesses owned by disabled veterans. From the information collected, Outreach developed and implemented a tracking tool to capture certification information about these firms. Of the 159 firms, 85 currently have a DVBE certification and 74 do not.

In May 2019, Outreach developed a survey for the 74 disabled veteran owned firms who are not yet certified, to capture potential reasons why they had not yet attained their DVBE certification. The survey was sent to these non-certified DVBE firms in June 2019.

As of July 2019, minimal survey responses have been received. Over half of respondents indicate that they have applied for DVBE certification, but have not yet been approved, or are planning on applying soon. Outreach continues to monitor the number of pending DVBE applications and the number of DVBE firms newly certified.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

The DGS Outreach Team is currently utilizing tools at DVBE events to encourage potential DVBE firms to get more information about certification. The outreach team is also collecting and tracking businesses owned by disabled veterans that attend the outreach events. In addition, certification data is being collected and will be reviewed to determine whether these businesses obtain their certifications. Furthermore, the outreach team will be collaborating with the California Department of Veterans Affairs (CalVet) to develop a survey that will be sent to DVBEs who received their certification last year. By January 1, 2020, the Outreach Team will analyze all available gathered data regarding DVBE participation at outreach and certification events. The data will be assessed for the effectiveness of the 2019 outreach efforts and for increasing the number of DVBE firms that become certified. OSDS will use the assessment to create an Outreach Plan for 2020 that will better address the needs of state departments and the DVBE community.

Estimated Completion Date: January 2020
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #17 To: Veterans Affairs, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, CalVet should conduct periodic surveys of businesses owned by disabled veterans that attended its outreach events but chose not to become DVBE certified to determine the reasons for not applying for the certification. The department should use this information to improve its outreach and any other areas of the program.

Annual Follow-Up Agency Response From October 2022

In an additional effort to measure the effectiveness of our outreach efforts, CalVet developed and implemented a Veteran Owned Business survey to better understand veteran owned businesses that do not have a DVBE certification, and assist with targeted outreach efforts to appeal to the benefits of DVBE certification. CalVet gathered and analyzed data sets through the U.S. Small Business Administration and other community-based organizations to gain greater access to California veteran business owners. Upon access to the data sets, CalVet released a survey to 1699 Non-DVBE veteran owned businesses to determine the reason they chose not to become certified. CalVet additionally offered multiple avenues of assistance in becoming a certified DVBE, as well as assistance with any presumed barriers in that process. Survey results indicated a strong need to continue to educate veterans on the DVBE program, the state contracting process, and to continue our concentrated outreach efforts on the procurement of goods and services. In addition, CalVet and DGS have reoccurring monthly meetings wherein the DGS Certification Unit cross references CalVet's outreach registration list to determine if business owners do in fact become certified after attended CalVet's outreach events.

Completion Date: October 2022

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2021

Through the executed interagency agreement with DGS, CalVet has gained greater access to certification and procurement data. Although this has allowed for greater resolution toward the impact of DVBEs who are newly certified, it does not provide the necessary information to discover disabled veteran business owners who did not become certified DVBEs. Due to this, CalVet has implemented additional avenues through the Request for Response as well as a foundation for data collection through post event surveys. Despite these efforts, veterans meeting the criteria indicated in this recommendation have not provided adequate responses to allow for meaningful program development. CalVet will continue to explore data sets through the U.S. Small Business Administration, the Governor's Office of Business and Economic Development, and other community based organizations to gain greater access to this population to improve its outreach processes.

Estimated Completion Date: July 1, 2022

California State Auditor's Assessment of Annual Follow-Up Status: Pending

Annual Follow-Up Agency Response From October 2020

The new IA does include the necessary data sharing to allow CalVet to perform surveys and analysis as recommended through this audit. Due to constraints related to COVID-19, CalVet will begin receiving this data no earlier than November 01, 2020. The agreement also provides the resources necessary to disseminate efforts down to the DVBE community through a revamped DVBE Advisory Council that will act as bridge between DGS/CalVet and the DVBE community as a whole. Key areas of investigation for the Council will included topics such as DVBE certification challenges and process improvements affecting DVBE recruitment.

Documented Evidence:

1) Interagency Agreement (executed)

2) CalVet Advisory Council Process Draft

3) DVBE AC Process Flow Draft

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

The new IA will include the necessary data sharing to allow CalVet to perform surveys and analysis as recommended through this audit. CalVet currently receives no DVBE certification or procurement data from DGS. The agreement will also provide the resources to disseminate research efforts down to the DVBE community through the DVBE Advisory Council. The Council will consist of DVBE-focused organizations and other veteran services groups that will act as bridge between DGS/CalVet and the DVBE community as a whole. Standing agenda items for the Council will includes topics such as DVBE certification challenges and process improvements affecting DVBE recruitment.

Estimated Completion Date: 12/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

CalVet surveys the outreach event attendees to evaluate the efficacy of the event. In particular, CalVet asks whether business owners applied for DVBE certification. CalVet is still evaluating the responses and intends to follow up with business owners deciding not apply for certification. CalVet will track the reasons cited by these attendees for not pursuing a DVBE certification and will use this information to improve future outreach events.

CalVet continues to evaluate survey responses. However, based on feedback during the events, two barriers to DVBE certification were immediately identified:

-First, some disabled veterans had not yet filed a disability claim with the VA. In response, CalVet now has a County Veteran Service Office representative present at all outreach events to assist veterans with claims.

-Second, attendees identified a need for more accessible websites with a clear process for becoming a DVBE. In response, CalVet updated our website to include new user-friendly links to a master contact list of all awarding agency DVBE advocates, upcoming procurement opportunities, and a guide on "How to do Business with the State of California."

Estimated Completion Date: December 27, 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. CalVet reviewed past outreach events to learn why veteran-owned business chose not to be DVBE certified. The feedback noted that DGS and CalVet resources and websites were confusing. CalVet's updated website has new DGS information, improved links, and contact information. CalVet has been tracking DVBE calls and requests since March and is now better able to help the veteran population get DVBE certified.

The next focus will measure outreach to veteran businesses that opt out of DVBE certification. The outreach survey will be developed to cover questions on veterans' past state contract history, the challenges in obtaining a DVBE certification, whether DVBE certification is beneficial and whether CalVet outreach events are effective. A report will be generated for those attendees that are not DVBE certified. A 30/60/90-day report will be developed to survey the reasoning for not obtaining a DVBE certification. These findings will help determine future outreach methods.

Estimated Completion Date: October 31, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #18 To: General Services, Department of

To ensure that its outreach efforts are effective and result in a greater number of DVBE firms available that can provide the necessary goods and services awarding departments are seeking, General Services should conduct periodic surveys of businesses owned by disabled veterans that attended its outreach events but chose not to become DVBE certified to determine the reasons for not applying for the certification. The department should use this information to improve its outreach and any other areas of the program.

1-Year Agency Response

Since January 2019, Outreach has collected a list of disabled veteran owned businesses at each outreach event. In May 2019, Outreach cross referenced the list of disabled veteran owned businesses with certification data and compiled a list of all firms that had not yet obtained their certification. In June 2019, Outreach sent out a survey to determine the reasons why these firms had not yet attained their DVBE certification.

As of July 2019, minimal survey responses have been received. Of the responses submitted, over half of respondents indicated that they had applied for DVBE certification, but had not yet been approved or were planning on applying soon.

Roughly a quarter of respondents indicated their firm does not meet DVBE eligibility requirements and the remaining quarter indicated they felt there was too much paperwork required to complete the application.

Additional feedback indicates the majority of respondents believe the best way to improve the program would be to simplify the application process. The results of the survey provide Outreach with information to be used in future outreach change efforts in training, education and information sharing with the DVBE community.

To address the survey responses and suggestions we received, the following actions have been taken:

- DVBE marketing materials and How to do Business with the State brochure have been updated and provide additional resources to simplify the certification process as well as clarify eligibility requirements.

- Continued our participation at DVBE outreach events, giving overviews and one-on-one guidance to potential DVBE firms.

- Continued to partner with DVBE-focused strategic partners to strengthen knowledge in local communities about the DVBE program.

- Corrected broken web links and system defects that are encountered.

- Outreach continues to monitor the number of pending DVBE applications and the number of DVBE firms newly certified.

Completion Date: December 2019
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

Since January 2019, Outreach has collected a list of disabled veteran owned businesses at each outreach event. In May 2019, Outreach cross referenced the list of disabled veteran owned businesses with current certification data and compiled a list of all firms that had not yet obtained their certification. To date Outreach has collected data from 159 potential suppliers of which 74 are not certified. In May 2019, Outreach developed a survey to determine the reasons why these firms had not yet attained their DVBE certification. The survey was out in June 2019 to those not certified.

Outreach continues to monitor the number of pending DVBE applications and the number of DVBE firms newly certified.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Partially Implemented

General Services provided documentation that it surveyed businesses that attended its outreach events but chose not to become DVBE certified. However, it did not provide any documentation demonstrating that it has taken steps to improve its outreach and any other areas of the program based on the survey responses. Until it provides such documentation, we will report the status of this recommendation as partially implemented.

Auditee did not substantiate its claim of full implementation

60-Day Agency Response

At each outreach event OSDS attends, the Outreach Team requests all potential or current DVBE firms to provide their contact information, so that this information can be tracked in our internal DVBE database. By October 1, 2019, all DVBE attendees who provided contact information at an outreach event will be sent a survey that will assist in analyzing the goals of DVBE firms. All survey responses will be gathered and analyzed, allowing the Outreach Team to identify any reasons why DVBEs choose not to get certified, or possible motivations for getting certified without a desire to contract with the state. The Outreach Team will use this information to adjust strategies for reaching DVBE firms in the 2020 Outreach Plan which will be completed by January 1, 2020.

Estimated Completion Date: December 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #19 To: Veterans Affairs, Department of

Until the Legislature amends the law to transfer its responsibility for assisting underachieving departments to General Services, CalVet should develop and follow policies and procedures to identify and assist awarding departments that fail to meet, or are at risk of not meeting, the 3 percent DVBE participation goal.

Annual Follow-Up Agency Response From October 2021

CalVet has developed and implemented policies and procedures to identify and assist awarding departments that fail to meet, or at risk of not meeting the 3 percent DVBE goal. These procedures will be followed on an established periodic basis until the legislature amends the law to transfer responsibility for assisting underachieving departments to the DGS.

Completion Date: September 2021

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

Annual Follow-Up Agency Response From October 2020

With the full staffing of the two DVBE staff members on October 1, 2020, and the recent release of the DGS Consolidated Annual Report FY18-19, CalVet will begin the process of developing process and procedures to identify and assist awarding departments who fail to meet their DVBE goal.

Documented Evidence:

Duty Statements (signed) for the two DVBE employees

Estimated Completion Date: June 30, 2021

California State Auditor's Assessment of Annual Follow-Up Status: Pending

1-Year Agency Response

The establishment of an IA with DGS will provide CalVet the means to develop and follow policies and procedures to identify and assist awarding departments who fail to meet their DVBE goal. CalVet wants to reiterate that our internal audits and review team will be an integral part in ensuring full and timely implementation of the recommendations provided by the California State Auditor.

Estimated Completion Date: 08/2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

6-Month Agency Response

AB 1809 (Cervantes, 2019), which is moving through the legislative process, would transfer responsibility for assisting underachieving departments to DGS. Meanwhile, CalVet continues to work with underperforming departments, as recommended.

Estimated Completion Date: December 27, 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

CalVet agrees with this recommendation. CalVet agrees with the auditor's report that states on page 27: "Considering its expertise, resources, and role in the DVBE program, General Services is better suited than CalVet to assist underachieving departments. General Services agreed with this assessment. Such a shift in the departments' responsibilities would also allow CalVet to focus primarily on outreach activities."

Meanwhile, CalVet is in the developmental phase of drafting policies and procedures to identify and assist awarding departments that fail to meet, or are at risk of not meeting, the 3 percent DVBE participation goal. Upon completion of drafting the policies and procedures, CalVet intends to develop Standard Operating Procedures (SOP) for the Awarding Departments. The SOP will outline checks and balances for awarding departments that will include a survey sampling process to evaluate DVBE subcontractors and evaluate prime contractors' DVBE usage performance.

Additionally, CalVet will take an active role to assist DGS to identify and investigate DVBE program abuse. CalVet has taken measures to update our website to include a "DVBE Fraud and Abuse" intake process and has adopted an internal tracking system to ensure that DVBEs receive a response. The updated website has contact information for both CalVet and DGS. Both departments will collaborate in tracking the calls and emails on DVBE program abuse.

Supporting documents will be made available upon request.

Estimated Completion Date: December 27, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #20 To: General Services, Department of

To minimize the occurrence of program abuse and ensure that program abuse cases are handled appropriately and consistently, General Services should remind the awarding departments about preventive measures, including contract award notification to all subcontractors listed on the winning contractor's bid and providing the subcontractors with the name of the prime contractor, the role of the DVBE firm in the contract, the amount of the contract dollars designated for each subcontractor, and contact information that subcontractors can use if they encounter any issues on the project.

Annual Follow-Up Agency Response From October 2020

As of June 2020, the Department of General Services (DGS) has fully implemented this recommendation. In January 2020, DGS issued a statewide Broadcast Bulletin P-01-20 announcing a Program Guide location on the DGS website.

The Program Guide includes information to awarding departments on recommended practices for preventing small business (SB) and Disabled Veteran Business Enterprise (DVBE) program abuse, program violations and sanctions, commercially useful function (CUF), DVBE substitution requirements and procedures, and what constitutes SB and DVBE unlawful activities.

Between April and August 2019, DGS held six training sessions for procurement officials and advocates regarding SB/DVBE program abuse recognition, prevention and reporting.

On June 15, 2020, DGS issued a statewide communication through Broadcast Bulletin F-03-20 on the implementation of the Fiscal Information System for California's (FI$Cal) generated notification to prime contractors and all certified SB/DVBE subcontractors listed on all state contracts and purchase orders (PO) with SB and DVBE participation.

The notification provides the SB and DVBE subcontractors with the awarding department and prime contractor information, contract description, terms, awarded amount, and the percent of the award to be performed on the contract by each listed subcontractor. It also guides them to contact the prime contractor, includes a link to the DGS SB and DVBE advocate directory, and the DGS Office of Small Business and DVBE Services (OSDS) Program Compliance email address SB.DVBECompliance@dgs.ca.gov, so firms can report any complaints of non-compliance or fraud allegations regarding SB/DVBE participation on state contracts.

Completion Date: June 2020

California State Auditor's Assessment of Annual Follow-Up Status: Fully Implemented

1-Year Agency Response

DGS issued a January 2020 Broadcast Bulletin to all state departments informing all Purchasing Authority Contacts, Procurement and Contracting Officers and SB/DVBE Advocates of the Program Guide location on the DGS website at:

https://www.dgs.ca.gov/PD/Services/Page-Content/Procurement-Division-Services-List-Folder/File-a-Complaint-to-OSDS#@ViewBag.JumpTo

DGS will be posting important information regarding SB/DVBE program abuse in this Program Guide section. The section includes information on recommended practices, program violations and sanctions, commercially useful function (CUF), as well as DVBE substitution requirements and procedures.

Broadcast Bulletins are published on DGS website at:

https://www.dgs.ca.gov/PD/Resources/Page-Content/Procurement-Division-Resources-List-Folder/Broadcast-Bulletins#@ViewBag.JumpTo

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Pending

The documentation on General Services' website does not include information related to reminding departments about taking preventive measures to minimize the occurrence of program abuse as specified in our recommendation.

Auditee did not substantiate its claim of full implementation

6-Month Agency Response

DGS is providing guidance to State Departments regarding preventative measures including notification of subcontractors when they are named as DVBE subcontractors, as well as notifying them of their rights and responsibilities. These preventative measures are part of the Awarding Department Comprehensive Program Abuse Training. This training and materials are being posted on the DGS website (https://www.dgs.ca.gov/PD/Resources/Page-Content/Procurement-Division-Resources-List-Folder/SB-DVBE-Program-Violations-and-Sanctions), as well as included in handouts during training.

Completion Date: August 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Partially Implemented

The documentation that General Services provided included a document titled "Recommended Practices", which included the practice of notifying the prime contractor, at the time of the contract award, with a copy to the DVBE subcontractor. However, the documentation does not clearly support that General Services provided these "Recommended Practices" to all awarding departments. Until General Services can demonstrate that it has provided this information to all awarding departments, we will continue to report this recommendation as partially implemented.

Auditee did not substantiate its claim of full implementation

60-Day Agency Response

DGS is currently implementing guidance on preventative measures for state departments that include notification of subcontractors, as well as notifying them of their rights and responsibilities. Additional reminders will be included in the training materials for upcoming Awarding Department Comprehensive Program Abuse Training scheduled to begin in April 2019 and completed by August 2019. This training and materials will be posted on the DGS website as well as included in handouts during training.

Estimated Completion Date: August 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #21 To: General Services, Department of

To minimize the occurrence of program abuse and ensure that program abuse cases are handled appropriately and consistently, General Services should develop procedures for awarding departments to help them identify whether a complaint constitutes program abuse, document and track all complaints, and take appropriate steps when investigating program abuse complaints.

6-Month Agency Response

DGS is including best practice procedures and documents in the Awarding Department Comprehensive Program Abuse Training. Training includes how to identify program abuse allegations and distinguish these from other complaints. The training material includes documentation, reporting, and processing of these program abuse allegations. These procedures and training documents are being posted to the DGS website as well as included in handouts during training.

Completion Date: August 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DGS is currently in the process of updating its best practices documents and developing procedures to aid state departments in identification of program abuse vs other complaints as well as tracking and responding to these complaints. Once these procedures are finalized, they will be included in the training materials for the Awarding Department Comprehensive Program Abuse Training scheduled to begin in April 2019 and end in August 2019. This training and materials will be posted on the DGS website as well as included in handouts during training.

Estimated Completion Date: August 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #22 To: General Services, Department of

To strengthen the enforcement of DVBE laws, regulations, and guidelines, General Services should track program abuse complaints, including the type of program abuse, how it was reported or discovered, and the dates specific actions were taken on the case. This tracking will enable the awarding departments to identify weaknesses within their processes and provide insight into how to address these issues, improving the DVBE program overall.

6-Month Agency Response

DGS has completed development of its comprehensive and consolidated tracking of complaints as recommended. The goal was a single location to get information regarding the number of complaints, types, how they are reported and dates specific actions were taken.

Completion Date: June 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DGS is developing and updating its comprehensive and consolidated tracking of complaints as recommended. The goal is a single location to get information regarding the number of complaints, types, how they are reported and dates specific actions were taken. Testing of format is planned for April 2019 and completed by June 2019.

Estimated Completion Date: June 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #23 To: Transportation, Department of

To ensure that DVBE participation data are reported accurately and consistently, Caltrans should implement or strengthen a review process to ensure that DVBE participation amounts entered into its data systems or FI$Cal are accurate. This review process should include verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractor or subcontractor for high-value contracts that include DVBE participation.

6-Month Agency Response

This recommendation has been fully implemented as the first quarterly review was completed in May 2019. The top 10 high-dollar contracts were reviewed against seven criteria developed to ensure compliance with the DVBE Program. All 10 contracts were 100 percent compliant for all seven criteria.

Caltrans' DPAC is in the process of replacing CATS with a system (CATSII) more capable of capturing, tracking, and reporting accurate data. We expect CATSII to be up and running September 2020.

Completion Date: May 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

Caltrans Division of Procurements and Contracts (DPAC) has revised the Contract Officer/Analyst Section Peer Review Checklist to require that managers and peer reviewers verify the data entered into the Contracts Administrative Tracking System (CATS). DPAC also developed procedures and a checklist to perform quarterly reviews of high-dollar contracts to confirm DVBE certifications and verify that data was entered correctly in CATS. DPAC communicated both of these changes to their staff via emails.

DPAC continues working on the Request For Information for CATSII development and implementation.

Corrective action partially implemented; the first quarterly review will be completed by May 1, 2019.

Estimated Completion Date: January 20, 2021
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #24 To: Corrections and Rehabilitation, Department of

To ensure that DVBE participation data are reported accurately and consistently, Corrections should implement or strengthen a review process to ensure that DVBE participation amounts entered into its data systems or FI$Cal are accurate. This review process should include verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractor or subcontractor for high-value contracts that include DVBE participation.

1-Year Agency Response

Due to portal size limitations, please refer to CAP submission via email.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

Corrections provided policies it implemented for a secondary review process. It also provided documentation to show secondary review its staff conducted to verify, on a sample basis, the amounts awarded to, and the certification status of, the DVBE contractors or subcontractor for high-value contracts that include DVBE participation.

6-Month Agency Response

Create 810 Workgroup made up of key stakeholders: Meetings have been held on 11/30/18,1/18/19,3/5/19,3/14/19,and 5/10/19.Meetings will continue bimonthly through the fiscal year,then as needed until all issues/requests are resolved.PCR 2570-Develop Minority Indicator Report to cross reference SAP entries and 810 Large Report.Partially Implemented.Estimated completion: 9/15/19.Currently in development and pending further review and testing by the 810 workgroup.PCR 2617-810Enhancements-adding Fund Year to report criteria in order to report by each Fiscal Year (multi-year contracts).Partially Implemented.Estimated completion: 9/15/19.Currently in development and pending further review of workgroup.POP:PCR 2617.Revise and update the Contracting Guidelines (Non-IT): Partially Implemented,estimated completion date of 9/15/19.In development: Contract Guidelines and Proof of training/implementation.Revisions to the Contract Manager's Handbook will be made to include additional instruction regarding DVBE participation input for Service and Expense Orders and POs issued in BIS.Partially Implemented with estimated completion date of 9/15/19.In development:Contract Manager's Handbook and Proof of training/implementation.

Develop written procedures for data verification/reconciliation report process to ensure accuracy in data reporting.Partially Implemented,estimated completion date of 9/15/19.In development:

PCR 2570.In development and pending further review and testing by the 810 workgroup.Revise/update the IT Contracting Guidelines:Partially Implemented with estimated completion date of 8/15/19.Met with new ITAU SB/DVBE Advocate to make necessary changes to processes,procedures,update handbook,make changes to boilerplate information that vendors receive,create a secondary review process to validate numbers,and train staff.Once handbook processes,and procedures are updated,we will be discussing/training at our biweekly team meetings.Estimated completion 9/15/19.

Estimated Completion Date: September 2020
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

Create 810 Workgroup made up of key stakeholders: Office of Business Services' Contract Management Branch and Procurement Section, California Correctional Health Care Services, Facilities Planning Construction Management, Accounting, and the Statewide SB/DVBE Advocate, to identify areas in need of correction within SAP to improve accuracy and to further automate reporting.To discuss recommendations and create a corrective action plan to ensure CDCR is reporting accurate data, strengthening the review process of verification of amounts awarded and the certification status of the DVBE contractor and/or subcontractor for high value contracts that include DVBE participation.

Production Change Requests (PCR):

PCR 1585 - Auto-populate Vendor Master Data in order to reduce keying errors, and provide additional oversight of SB/DVBE certification.

PCR 2570 - Develop Minority Indicator Report to cross reference SAP entries and 810 Large Report.

PCR 2617 - 810 Enhancements - adding Fund Year to report criteria in order to report by each Fiscal Year (multi-year contracts).

Revise and update the Contracting Guidelines (Non-IT) which is provided to contract analysts that detail instruction on the contract process. Additional instruction will be added to ensure that DVBE participation is entered correctly when contracts and purchase orders (POs) are issued in BIS.Revisions to the Contract Manager's Handbook will be made to include additional instruction regarding DVBE participation input for Service and Expense Orders and POs issued in BIS. Develop written procedures for data verification/reconciliation report process to ensure accuracy in data reporting.The Procurement Operations and Delegation Oversight (PODO) Unit has developed a process for review of non-IT goods procurements to ensure accuracy in data reporting and to identify any discrepancies in the 810 Report.

Estimated Completion Date: July 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #25 To: Motor Vehicles, Department of

To ensure that DVBE participation data are reported accurately and consistently, DMV should implement or strengthen a review process to ensure that DVBE participation amounts entered into its data systems or FI$Cal are accurate. This review process should include verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractor or subcontractor for high-value contracts that include DVBE participation.

60-Day Agency Response

DMV concurs with the recommendation. The DMV takes the DVBE program very seriously and continues to be committed to exceeding the three-percent goal for state governmental agencies. Additional training was conducted to ensure staff are following proper procedures. Further, beginning February 2019, DMV is implementing a quarterly process to check Small Business/Disabled Veteran Business Enterprise amounts awarded and certification status for high-value contracts exceeding $100,000. In the interim, to assure program management that the error found during the audit is not a widespread problem, DMV's Internal Audits Office randomly sampled 20 contracts of varying amounts awarded and found the DVBE calculations were correct on all 20 contracts.

Completion Date: February 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Fully Implemented

Recommendation #26 To: General Services, Department of

To ensure that DVBE participation data are reported accurately and consistently, General Services should implement or strengthen a review process to ensure that DVBE participation amounts entered into its data systems or FI$Cal are accurate. This review process should include verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractor or subcontractor for high-value contracts that include DVBE participation.

1-Year Agency Response

To ensure that DVBE participation data is reported accurately and consistently in its data systems or FI$Cal, DGS' Office of Business and Acquisition Services (OBAS) has taken the following steps in implementation of a review process which includes verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractors and subcontractors:

- Finalized the Small Business (SB) and DVBE Advocate Desk Manual.

- Finalized and tested procedures related to CAR reporting and contract verification (incorporated within the Desk Manual) in August of 2019.

Completion Date: January 2020
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

To ensure that DVBE participation data are reported accurately and consistently in its data systems or FI$Cal, DGS' Office of Business and Acquisition Services (OBAS) have taken the following steps toward implementation and will be fully implemented by October 2019:

- Continues to outline the Small Business (SB) and DVBE Advocate Desk Manual.

- Draft procedures related to CAR reporting and contract verification created, revisions still in progress.

- Plan to test procedures on CAR report due August 1, 2019.

Estimated Completion Date: October 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

- Attended the Consolidated Annual Report (CAR) training offered by OSDS on 03/06/19. Information obtained on how to run new CAR reports and new reporting requirements.

- Created a template for tracking sample basis contract verification. This template is still in draft format. Additional instructions and requirements will be added as procedures are developed.

- Began outlining the Small Business (SB) and DVBE Advocate Desk Manual. Procedures related to CAR reporting and contract verification.

Estimated Completion Date: October 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #27 To: Public Health, Department of

To ensure that DVBE participation data are reported accurately and consistently, Public Health should implement or strengthen a review process to ensure that DVBE participation amounts entered into its data systems or FI$Cal are accurate. This review process should include verification, on a sample basis, of the amounts awarded to, and the certification status of, the DVBE contractor or subcontractor for high-value contracts that include DVBE participation.

6-Month Agency Response

CDPH enhanced its Contracts and Purchasing System (CAPS) training for new and current staff, in both Program Support Branch (PSB) and across CDPH Programs, to reiterate the importance of completing all data in CAPS accurately, with an extra focus on subcontracting issues and compliance.

In order to strengthen its secondary review process of procurements, CDPH updated its checklists and routing slips used in the Contracts and Purchasing Services Section (CPSS) for competitive solicitations.

CDPH's Small Business (SB)/DVBE Advocate (the Advocate) provided specific DVBE language to be used in solicitations.

CDPH provides Consolidated Annual Report (CAR) training regarding SB and DVBE participation. The training is presented annually by either the Advocate or training staff from the Policy, Training and Administrative Services Unit. These training materials have been incorporated into the PSB Training Series for programs.

The Advocate runs quarterly reports from CAPS and FI$Cal to verify accuracy and resolve discrepancies with internal staff in order to improve data accuracy in the CAR.

CDPH updated the Purchasing Manual section on SB/DVBE Participation to a more user-friendly chapter based on DGS accreditation requirements.

Completion Date: July 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

CDPH will enhance its Contracts and Purchasing System (CAPS) training for new and current staff, in both Program Support Branch (PSB) and across CDPH Programs, to reiterate the importance of completing all data in CAPS accurately, with an extra focus on subcontracting issues and compliance. (By 4/30/19)

In order to strengthen its secondary review process of procurements, CDPH will update checklists and routing slips used in the Contracts and Purchasing Services Section (CPSS) for competitive solicitations. (By 4/30/19)

CDPH's Small Business (SB) /DVBE Advocate (the Advocate) will review draft solicitation documentation and provide specific DVBE language to be used in solicitations. (By 5/31/19)

CDPH will annually provide Consolidated Annual Report (CAR) training regarding SB and DVBE participation. The training will be presented by either the Advocate or training staff from the Policy, Training and Administrative Services Unit. These training materials will also be incorporated into the PSB Training Series for programs. (By 6/30/19)

The Advocate will run quarterly reports from CAPS and FI$Cal to verify accuracy and resolve discrepancies with internal staff in order to improve data accuracy in the CAR. (By 6/30/19)

CDPH will update the Purchasing Manual section on SB/DVBE Participation to a more user-friendly chapter based on DGS accreditation requirements. (By 6/30/19)

Estimated Completion Date: June 30, 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #28 To: General Services, Department of

Until it begins generating DVBE activity reports using FI$Cal, General Services should issue a policy to require awarding departments to implement or strengthen a secondary review process to ensure that the DVBE activity reports are accurate and supported by departments' data systems.

1-Year Agency Response

DGS updated the State Contracting Manual (SCM) with the policy described below and issued Broadcast Bulletin #P 13-19 on December 20, 2019 to notify state departments of the new policy.

The revised SCM policy and procedures require state agencies to:

- Implement a review process to ensure that the Small Business (SB) and Disabled Veteran Business Enterprise (DVBE) participation data, including for Microbusiness (SB/MB), Small Business for the purpose of Public Works (SB-PW) and Nonprofit Veteran Services Agency (NVSA), is accurate.

- Verify on a sample basis, the amounts awarded to and the certification status of the SB, SB/MB, SB-PW, DVBE or the NVSA contractor or subcontractors for high value contracts that included SB or DVBE participation.

- Provide by request a written statement signed by the Procurement and Contracting Officer (PCO) confirming completion of the review of the CAR.

Broadcast Bulletins are published on DGS website at:

https://www.dgs.ca.gov/PD/Resources/Page-Content/Procurement-Division-Resources-List-Folder/Broadcast-Bulletins#@ViewBag.JumpTo

Completion Date: December 2019
Response Date: February 2020

California State Auditor's Assessment of 1-Year Status: Fully Implemented

6-Month Agency Response

DGS incorporated into all eight CAR training sessions held between February and May 2019 (February 26, March 6, 13 and 20, April 23 and 24, May 3 and 8), information requiring awarding departments to do the following:

- Develop, implement or strengthen a secondary review process to ensure that the DVBE activity reports prepared by staff are accurate and supported by documentation.

- Verify on a sample basis, the amount awarded to and the certification status of the DVBE contractor or subcontractor on a contract that includes DVBE participation.

In addition, DGS will update the reporting sections of the State Contracting Manuals (SCM) to include these reporting requirements by September 2019.

Estimated Completion Date: September 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Pending

60-Day Agency Response

On February 6, 2019, DGS issued Broadcast Bulletin E-03-19 for training on reporting Consolidated Annual Report (CAR) in FI$Cal. DGS incorporated in its February-March 2019 (CAR) training the requirements for departments to develop and strengthen a secondary review process of their reports, and to verify on a sample basis the amount awarded to and the certification status of the DVBE contractor or subcontractor. Training session are scheduled to begin in February 2019 and end in May 2019.

In addition, DGS is in the process of updating the State Contracting Manuals (SCM) to include these reporting requirements. Once the updates are completed by May 2019, DGS will notify state departments through a broadcast bulletin.

Estimated Completion Date: May 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

Recommendation #29 To: General Services, Department of

To ensure that it can create accurate annual DVBE participation reports, General Services should complete the programming necessary to include DVBE subcontractor information by June 30, 2019.

6-Month Agency Response

DGS completed the Consolidated Annual Report (CAR) programming and development efforts of the CAR functionality implementation into FI$Cal in June 2019. DGS will continue to work with state departments post-implementation to identify any problems and resolve any production defects. Please see FI$Cal/SCPRS User's Guide to CAR Reporting (May 2019).

Completion Date: June 2019
Response Date: August 2019

California State Auditor's Assessment of 6-Month Status: Fully Implemented

60-Day Agency Response

DGS completed the Consolidated Annual Report (CAR) programming and development efforts of the CAR functionality implementation into FI$Cal. DGS is currently working on the final testing of the CAR development and the full implementation into production. Data validation is estimated to be completed in May 2019. DGS will continue to work with FI$Cal post-implementation to identify any problems and resolve any production defects.

Estimated Completion Date: June 2019
Response Date: April 2019

California State Auditor's Assessment of 60-Day Status: Pending

All Recommendations in 2018-114

Agency responses received are posted verbatim.